[00:00:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3859.000000 [00:02:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:04:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:08:31] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 318185 MB (5% inode=34%): [00:09:02] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:10:44] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3801 [00:24:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:36:36] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:18] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40394 MB (9% inode=99%): [00:51:18] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 320106 MB (6% inode=34%): [00:53:19] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:53:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:56:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:58:33] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [01:00:24] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3820.000000 [01:03:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:05:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:09:47] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 59223 MB (14% inode=99%): [01:10:17] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:11:00] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4013 [01:25:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:36:51] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:34] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 319510 MB (6% inode=34%): [01:53:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:54:35] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:57:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:58:57] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [02:00:58] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4899.000000 [02:03:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:05:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:07:41] /sql on thyme is WARNING: DISK WARNING - free space: /sql 185272 MB (19% inode=99%): [02:10:52] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:11:33] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4847 [02:24:25] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 318272 MB (5% inode=34%): [02:25:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:37:27] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:55:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:58:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:59:51] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [03:01:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4804.000000 [03:04:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:06:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:11:04] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:11:46] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4820 [03:24:47] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317621 MB (5% inode=34%): [03:25:48] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:37:39] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:55:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:58:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:00:13] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [04:02:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5036.000000 [04:04:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:06:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:11:28] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:12:09] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4991 [04:24:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317569 MB (5% inode=34%): [04:26:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:38:32] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:55:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:56:04] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:59:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:00:47] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [05:02:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4791.000000 [05:05:08] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:07:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:12:20] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:13:01] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4822 [05:25:41] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317496 MB (5% inode=34%): [05:27:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:39:14] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:56:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:59:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:01:39] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [06:03:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4420.000000 [06:05:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:08:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:13:03] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4297 [06:13:34] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:17] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317410 MB (5% inode=34%): [06:28:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:39:29] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:55:10] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3594.000000 [06:55:51] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3562 [06:56:52] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:57:33] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:00:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:02:14] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [07:05:57] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:08:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:13:47] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:26:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317348 MB (5% inode=34%): [07:29:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:30:52] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36675 MB (9% inode=99%): [07:39:43] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:43:44] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [07:44:15] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24469 MB (5% inode=99%): [07:44:56] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2961.000000 [07:56:48] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2678 [07:57:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:58:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:01:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:02:52] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [08:06:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:08:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:14:45] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:27] MySQL slave on thyme is OK: Uptime: 563104 Threads: 19 Questions: 234405558 Slow queries: 85535 Opens: 44300 Flush tables: 1 Open tables: 3058 Queries per second avg: 416.274 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1800 [08:27:07] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317264 MB (5% inode=34%): [08:27:38] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1786.000000 [08:29:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:36:21] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 47502 MB (11% inode=99%): [08:40:32] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:23] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 34014 MB (8% inode=99%): [08:55:53] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:57:35] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:58:36] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:01:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:03:40] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [09:07:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:09:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:14:51] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:27:33] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317195 MB (5% inode=34%): [09:29:34] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:41:24] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:54:06] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40246 MB (9% inode=99%): [09:58:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:58:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:02:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:04:11] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [10:08:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:09:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:15:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:50] Does anyone know why SGE jobs all seem to be queued, nothing running? [10:16:25] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 57016 MB (14% inode=99%): [10:28:26] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317112 MB (5% inode=34%): [10:30:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:37:18] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 62429 MB (5% inode=98%): [10:42:18] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:31] hello all [10:48:03] DaB. * [Toolserver-announce] Clarification about Rule 9.4 (no-interwiki-bots) [10:56:10] [[Query service]] ! 10https://wiki.toolserver.org/w/index.php?diff=7417&oldid=6585&rcid=10038 * 160.45.234.215 * (+1) () [10:57:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.398926/1.95, alarm hl:tmp_free=24253M/100M, alarm hl:np_load_avg=0.362305/2.0, alarm hl:mem_free=1540.000000M/350M, alarm hl:available=0/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.398926/2.3, alarm hl:np_load_long=0.406738/2.5, alarm hl:cpu=40.800000/98, alarm hl:mem_free=1540.000000M/200M, [10:57:57] [[Query service]] ! 10https://wiki.toolserver.org/w/index.php?diff=7418&oldid=7417&rcid=10039 * 160.45.234.215 * (+34) () [10:58:15] [[Query service]] ! 10https://wiki.toolserver.org/w/index.php?diff=7419&oldid=7418&rcid=10040 * 160.45.234.215 * (-35) () [10:59:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:59:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:59:42] [[User talk:160.45.234.215]] N 10https://wiki.toolserver.org/w/index.php?oldid=7420&rcid=10041 * Dab * (+113) (Created page with "Hello,
please use the [[Sandbox]] or another testing-page ;-). --~~~~") [11:01:14] Sun Grid Engine execd on willow is OK: testqueue@willow disabled: medium-sol@willow disabled by calendar: longrun-sol@willow disabled by calendar [11:03:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:04:17] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [11:06:23] oh dear [11:06:59] closedmouth: rebooting willow as annouced [11:09:42] Sun Grid Engine execd on wolfsbane is CRITICAL: Connection refused by host [11:12:48] DaBPunkt, what's going on with SGE? No jobs seem to run for the last several hours? [11:14:01] RnB: no jobs were started with an runtime longer than 11 o'clock utc because of the announced reboot [11:14:03] Merlissimo: qstat shows there is something wrong with sge [11:14:10] ah ok [11:14:43] DaBPunkt: no execd [11:15:48] fixed for willow [11:17:18] I will be SO happy when we are free of these solaris-boxes.… [11:17:42] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:17:49] Merlissimo: Thanks. So all those jobs that are now queued will start sometime, since the reboot is done? [11:18:08] RnB: yes [11:18:48] * DaBPunkt hopes with some throttling… [11:18:55] more exacly: after maintenace calendar entry is expired [11:20:20] exec is back on ll hosts [11:20:25] +a [11:28:45] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 317093 MB (5% inode=34%): [11:29:54] submit is being unhappy [11:30:01] refuses ssh connection [11:30:15] ... and it is back again [11:30:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [11:35:01] DaBPunkt: sorry for the ping, but is there something wrong with https://jira.toolserver.org/browse/ACCAPP-507 :) ? [11:35:33] Melos: no sorry, I was just busy. [11:35:53] ok thanks [11:42:03] SMTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:45:03] SMTP on ortelius is OK: SMTP OK - 0.003 sec. response time [11:55:07] Alchimista * [Toolserver-l] Bottle [11:58:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:59:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:04:02] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [12:08:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:17:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 326074 MB (6% inode=34%): [12:28:42] DaBPunkt: around? [12:30:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [12:51:43] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [12:53:12] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1710.000000 [12:58:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:59:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:01:46] !replag [13:01:52] @replag [13:01:52] DarkoNeko: s1-user: 26m 17s [-]; s2-user: 3h 12m 32s [-] [13:04:03] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [13:08:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:09:15] @replag [13:09:15] DarkoNeko: s1-user: 24m 27s [-0.33 s/s]; s2-user: 3h 19m 56s [+1.00 s/s]; s3-rr-a: 47s [-]; s3-user: 47s [-] [13:09:23] uh oh [13:09:35] s2-user stopped [13:17:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 328706 MB (6% inode=34%): [13:30:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [13:59:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:59:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:04:03] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [14:04:07] Bryan Tong Minh * Re: [Toolserver-l] Bottle [14:07:53] /sql on thyme is WARNING: DISK WARNING - free space: /sql 183893 MB (19% inode=99%): [14:09:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:17:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:27:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 326046 MB (6% inode=34%): [14:30:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [14:59:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:59:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:04:03] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [15:09:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:17:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:27:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 325018 MB (6% inode=34%): [15:30:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [15:45:07] Mashiah Davidson * Re: [Toolserver-l] Clarification about Rule 9.4 (no-interwiki-bots) [15:59:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:59:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:04:03] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [16:09:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:18:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:27:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 324708 MB (6% inode=34%): [16:30:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [16:59:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:59:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:04:03] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [17:09:04] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:18:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 323159 MB (6% inode=34%): [17:31:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [17:48:43] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1985 [17:49:03] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1987.000000 [17:59:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:00:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:04:03] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [18:09:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:18:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 322093 MB (6% inode=34%): [18:32:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [18:48:43] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2822 [18:49:02] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2824.000000 [18:59:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:00:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:04:04] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [19:09:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:18:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 320786 MB (6% inode=34%): [19:32:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [19:44:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3619 [19:45:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3633.000000 [19:59:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:00:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:05:03] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [20:09:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:18:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:42] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 320533 MB (6% inode=34%): [20:32:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [20:44:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5471 [20:45:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5483.000000 [20:53:14] @replag [20:53:14] DarkoNeko: s1-rr-a: 16m 37s [-]; s1-user: 1h 32m 5s [+0.26 s/s] [20:59:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:00:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:05:03] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [21:09:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:18:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 319974 MB (6% inode=34%): [21:32:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [21:39:06] DaB. * Re: [Toolserver-l] Clarification about Rule 9.4 (no-interwiki-bots) [21:44:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6280 [21:45:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6288.000000 [21:59:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:00:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:05:03] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [22:09:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:18:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:27:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 319312 MB (6% inode=34%): [22:32:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [22:36:53] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 68365 MB (5% inode=99%): [22:41:03] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1888.000000 [22:41:44] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1923 [22:43:03] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [22:43:43] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2005.000000 [22:44:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7631 [22:45:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7644.000000 [22:59:04] nacht ts [22:59:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:00:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:05:03] ethernet 0/1/24 [ortelius.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/24:DOWN: 1 int NOK : CRITICAL [23:09:03] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:13:43] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 318234 MB (5% inode=34%): [23:18:43] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:20:26] Brooke: around? [23:32:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/network/rpc-100235_1/rpc_ticotsord:default svc:/application/jira:default [23:41:43] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2244 [23:44:11] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2352.000000 [23:44:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9149 [23:45:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9146.000000 [23:59:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default