[00:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 179387 [00:12:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:13:09] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 17.383 second response time [00:17:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64393 [00:18:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 179669.000000 [00:22:10] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:35:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:38:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:38:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:43:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64647.000000 [00:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 563214 MB (10% inode=47%): [01:00:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 180268 [01:08:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106027 MB (10% inode=98%): [01:09:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106124 MB (10% inode=98%): [01:17:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64821 [01:18:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 180488.000000 [01:22:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:22:59] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:29:49] @replag [01:29:49] Brooke: s1-rr-a: 17h 58m 5s [+0.03 s/s]; s1-user: 2d 2h 10m 17s [+0.21 s/s]; s3-rr-a: 60s [+0.00 s/s]; s3-user: 60s [+0.00 s/s]; s5-rr-a: 10s [-0.00 s/s]; s5-user: 10s [-0.00 s/s] [01:32:50] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:35:20] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:37:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:38:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:43:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64674.000000 [01:47:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40930 MB (10% inode=99%): [01:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 563169 MB (10% inode=47%): [02:00:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 180976 [02:04:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23213 MB (5% inode=99%): [02:06:09] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.843 second response time [02:09:09] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 273 bytes in 20.890 second response time [02:11:49] Load avg. on willow is WARNING: WARNING - load average: 16.21, 15.93, 13.61 [02:12:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43565 MB (10% inode=99%): [02:14:40] /sql on thyme is WARNING: DISK WARNING - free space: /sql 170458 MB (17% inode=99%): [02:14:49] Load avg. on willow is OK: OK - load average: 12.90, 14.98, 13.68 [02:17:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64205 [02:18:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 181284.000000 [02:35:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:38:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58825 MB (14% inode=99%): [02:38:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:39:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:43:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64074.000000 [02:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:57:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:59:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 561229 MB (10% inode=47%): [03:00:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 181822 [03:09:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:16:50] Load avg. on willow is WARNING: WARNING - load average: 15.12, 16.11, 14.66 [03:17:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63998 [03:18:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 181866.000000 [03:24:49] Load avg. on willow is OK: OK - load average: 12.75, 14.65, 14.68 [03:35:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:38:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:39:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:43:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64070.000000 [03:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:55:29] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107243 MB (11% inode=98%): [03:55:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107227 MB (11% inode=98%): [03:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 561333 MB (10% inode=47%): [04:00:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 182432 [04:09:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:09:49] Load avg. on willow is WARNING: WARNING - load average: 16.59, 16.65, 14.66 [04:17:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63993 [04:17:49] Load avg. on willow is OK: OK - load average: 12.75, 14.66, 14.76 [04:18:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 182658.000000 [04:28:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44189 MB (10% inode=99%): [04:35:20] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:39:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:39:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:43:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64057.000000 [04:46:49] Load avg. on willow is WARNING: WARNING - load average: 15.20, 15.34, 15.09 [04:47:00] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:47:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 53669 MB (13% inode=99%): [04:47:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106802 MB (10% inode=98%): [04:47:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106797 MB (10% inode=98%): [04:47:50] Load avg. on willow is OK: OK - load average: 13.81, 14.87, 14.93 [04:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 561268 MB (10% inode=47%): [05:00:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 183295 [05:09:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:17:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64436 [05:19:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 183621.000000 [05:21:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41640 MB (10% inode=99%): [05:24:49] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 54304 MB (4% inode=98%): [05:32:49] Load avg. on willow is WARNING: WARNING - load average: 12.82, 15.90, 14.73 [05:34:50] Load avg. on willow is OK: OK - load average: 10.79, 14.41, 14.33 [05:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:39:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:40:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:43:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64677.000000 [05:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:49:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24365 MB (5% inode=99%): [05:54:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 53628 MB (13% inode=99%): [05:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 561200 MB (10% inode=47%): [06:00:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 184118 [06:06:49] Load avg. on willow is WARNING: WARNING - load average: 22.46, 18.61, 15.94 [06:09:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:17:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64782 [06:19:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 184502.000000 [06:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:39:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:40:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:43:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64835.000000 [06:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 561093 MB (10% inode=47%): [07:00:50] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.44, 20.91, 19.40 [07:00:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:01:50] Load avg. on willow is WARNING: WARNING - load average: 20.62, 19.75, 19.08 [07:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 184950 [07:09:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:09:49] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.71, 22.18, 20.21 [07:17:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64709 [07:17:49] Load avg. on willow is WARNING: WARNING - load average: 16.02, 19.09, 19.96 [07:19:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 185113.000000 [07:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:40:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:40:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:43:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64907.000000 [07:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 560974 MB (10% inode=47%): [08:00:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 185629 [08:05:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44097 MB (10% inode=99%): [08:09:09] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:14:59] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 274 bytes in 8.670 second response time [08:17:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 65140 [08:17:50] Load avg. on willow is WARNING: WARNING - load average: 12.44, 14.43, 15.86 [08:19:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 185888.000000 [08:22:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 55250 MB (13% inode=99%): [08:28:49] Load avg. on willow is OK: OK - load average: 12.82, 13.62, 14.91 [08:36:18] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:40:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:44:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:44:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:46:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65262.000000 [08:48:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:54:10] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:54:59] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:50] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 104534 MB (10% inode=98%): [08:55:51] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 104534 MB (10% inode=98%): [08:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 560714 MB (10% inode=47%): [09:00:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 186026 [09:13:10] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:13:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 104497 MB (10% inode=98%): [09:17:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64957 [09:20:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 186184.000000 [09:29:00] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:29:10] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 104076 MB (10% inode=98%): [09:31:07] @replag [09:31:07] Merlissimo: s1-rr-a: 17h 59m 28s [+0.00 s/s]; s1-user: 2d 3h 52m 55s [+0.21 s/s]; s3-rr-a: 16s [-0.00 s/s]; s3-user: 16s [-0.00 s/s]; s6-rr-a: 1m 1s [+0.00 s/s]; s6-user: 1m 1s [+0.00 s/s]; s7-rr-a: 30s [+0.00 s/s]; s7-user: 30s [+0.00 s/s] [09:36:20] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:40:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:41:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:44:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:46:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64553.000000 [09:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:48:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 103947 MB (10% inode=98%): [09:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 560531 MB (10% inode=47%): [10:00:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:02:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 187119 [10:18:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64018 [10:20:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 187188.000000 [10:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:40:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:41:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:45:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63605.000000 [10:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:59:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 560381 MB (10% inode=47%): [11:00:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 187145 [11:14:41] hello [11:18:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63093 [11:20:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 187117.000000 [11:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:41:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:41:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:45:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62684.000000 [11:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:57:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 560157 MB (10% inode=47%): [12:00:06] DaB. * [Toolserver-l] FYI: Mailinglist-changes [12:00:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:02:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 186811 [12:18:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62604 [12:20:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 186773.000000 [12:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:41:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:41:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:45:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41528 MB (10% inode=99%): [12:45:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62770.000000 [12:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:54:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 56808 MB (14% inode=99%): [12:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559969 MB (10% inode=47%): [13:00:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:02:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 186476 [13:04:18] @replag [13:04:18] Dispenser: s1-rr-a: 17h 25m 23s [-0.16 s/s]; s1-user: 2d 3h 48m 7s [-0.02 s/s] [13:18:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62639 [13:20:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 186407.000000 [13:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:41:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:42:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:45:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62629.000000 [13:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559790 MB (10% inode=47%): [14:00:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:02:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 186479 [14:14:40] /sql on thyme is WARNING: DISK WARNING - free space: /sql 167995 MB (17% inode=99%): [14:16:06] Russell Blau * Re: [Toolserver-l] s1 replag, some observations [14:18:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 62569 [14:20:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 186587.000000 [14:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:41:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:42:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:45:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:46:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62686.000000 [14:48:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:59:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559650 MB (10% inode=47%): [15:00:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:02:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 186660 [15:14:50] Load avg. on willow is WARNING: WARNING - load average: 14.55, 15.73, 14.55 [15:18:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63215 [15:19:49] Load avg. on willow is OK: OK - load average: 13.45, 14.79, 14.54 [15:20:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 186663.000000 [15:36:18] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:41:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:42:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:45:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:46:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63726.000000 [15:49:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559473 MB (10% inode=47%): [16:00:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:02:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 187086 [16:11:50] APT on yarrow is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. run with -v for information. [16:12:49] APT on nightshade is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. run with -v for information. [16:18:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64044 [16:20:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 187299.000000 [16:36:20] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:41:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:42:06] @replag [16:42:06] Hello71: s1-rr-a: 17h 48m 1s [+0.10 s/s]; s1-user: 2d 4h 3m 41s [+0.07 s/s]; s2-user: 38s [+0.00 s/s]; s3-rr-a: 21s [+0.00 s/s]; s3-user: 21s [+0.00 s/s] [16:42:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:45:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:46:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64112.000000 [16:49:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 559052 MB (10% inode=47%): [17:01:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:02:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 187663 [17:06:50] Load avg. on willow is WARNING: WARNING - load average: 14.52, 15.31, 13.02 [17:11:50] APT on yarrow is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. run with -v for information. [17:12:50] APT on nightshade is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. run with -v for information. [17:14:50] Load avg. on willow is OK: OK - load average: 12.62, 14.82, 13.93 [17:18:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64282 [17:20:28] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 187840.000000 [17:22:52] → /wl [17:22:56] oopsie [17:24:50] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 58006 MB (4% inode=98%): [17:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:41:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:43:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:45:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:46:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64369.000000 [17:49:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558881 MB (10% inode=47%): [18:02:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 188181 [18:02:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:11:50] APT on yarrow is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. run with -v for information. [18:12:49] APT on nightshade is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. run with -v for information. [18:18:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64464 [18:20:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 188543.000000 [18:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:40:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40788 MB (10% inode=99%): [18:41:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:43:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:45:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:47:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64543.000000 [18:49:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:57:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:57:49] APT on yarrow is CRITICAL: Connection refused by host [18:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558733 MB (10% inode=47%): [19:02:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 189192 [19:02:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:04:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24215 MB (5% inode=99%): [19:10:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 53180 MB (13% inode=99%): [19:12:49] APT on nightshade is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. run with -v for information. [19:18:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64420 [19:21:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 189590.000000 [19:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:41:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:43:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:45:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:46:50] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:47:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64160.000000 [19:49:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:49:41] DaBPunkt : WOuld it be possible to raise the maxnumber of connections for the erfgoed account? [19:50:01] sure. Fil in a ticket [19:52:29] DaBPunkt: https://jira.toolserver.org/browse/TS-1443 [19:52:50] Would be very nice. We're hacking mobile here right now and things are not working :-( [19:52:59] NTP on yarrow is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset 0.029485 secs [19:57:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558590 MB (10% inode=47%): [20:01:59] NTP on yarrow is OK: NTP OK: Offset 0.072343 secs [20:02:30] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 190223 [20:02:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:11:26] Thank you DaBPunkt! [20:11:31] np [20:12:49] APT on nightshade is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. run with -v for information. [20:15:59] / on willow is WARNING: DISK WARNING - free space: / 21972 MB (20% inode=99%): [20:16:09] /tmp on willow is WARNING: DISK WARNING - free space: / 21972 MB (20% inode=99%): [20:18:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64068 [20:21:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 190624.000000 [20:27:02] DaBPunkt: Did alexz release his code under a free license? [20:27:58] tsDefaultLicense: GPLv3 [20:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:42:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:43:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:45:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:47:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64173.000000 [20:47:09] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 15.623 second response time [20:49:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:53:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40963 MB (10% inode=99%): [20:54:19] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558394 MB (10% inode=47%): [21:02:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 191182 [21:02:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:13:49] APT on nightshade is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. run with -v for information. [21:15:49] APT on yarrow is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. run with -v for information. [21:16:09] /tmp on willow is WARNING: DISK WARNING - free space: / 21904 MB (20% inode=99%): [21:16:59] / on willow is WARNING: DISK WARNING - free space: / 21904 MB (20% inode=99%): [21:18:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24407 MB (5% inode=99%): [21:18:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64123 [21:19:50] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:20:49] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:22:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 54062 MB (13% inode=99%): [21:22:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 191573.000000 [21:29:23] [[User:Nosy]] ! 10https://wiki.toolserver.org/w/index.php?diff=7439&oldid=7421&rcid=10096 * 92.206.156.14 * (+7) () [21:29:35] [[User:Nosy]] ! 10https://wiki.toolserver.org/w/index.php?diff=7440&oldid=7439&rcid=10097 * 92.206.156.14 * (-7) () [21:30:40] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 105342 MB (10% inode=98%): [21:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:42:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:44:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:45:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:46:05] Dr. Trigon * Re: [Toolserver-l] Quota [21:47:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64082.000000 [21:48:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 105339 MB (10% inode=98%): [21:49:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:54:19] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [21:57:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:59:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558251 MB (10% inode=47%): [22:02:29] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 192132 [22:02:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:11:09] /tmp on willow is OK: DISK OK - free space: / 34298 MB (32% inode=99%): [22:11:59] / on willow is OK: DISK OK - free space: / 34231 MB (32% inode=99%): [22:19:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64112 [22:22:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 192574.000000 [22:36:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:44:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:44:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 192970.000000 [22:45:55] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:46:15] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:47:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64008.000000 [22:49:16] Sun Grid Engine execd on yarrow is UNKNOWN: NRPE: Unable to read output [22:49:25] Sun Grid Engine execd on nightshade is UNKNOWN: NRPE: Unable to read output [22:49:25] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [22:49:35] Cluster on nightshade is UNKNOWN: NRPE: Unable to read output [22:49:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:50:06] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [22:50:06] SMF on nightshade is UNKNOWN: NRPE: Unable to read output [22:54:45] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [22:57:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:59:16] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 558121 MB (10% inode=47%): [23:03:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:03:16] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 193227 [23:19:05] Platonides * Re: [Toolserver-l] Quota [23:19:45] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63647 [23:26:05] SMF on yarrow is CRITICAL: Connection refused by host [23:26:15] Sun Grid Engine execd on yarrow is CRITICAL: Connection refused by host [23:26:25] Cluster on yarrow is CRITICAL: Connection refused by host [23:27:25] Cluster on yarrow is UNKNOWN: NRPE: Unable to read output [23:28:05] SMF on yarrow is UNKNOWN: NRPE: Unable to read output [23:28:15] Sun Grid Engine execd on yarrow is UNKNOWN: Error with qhost: OGS/GE 2011.11 [23:30:05] Platonides * Re: [Toolserver-l] s1 replag, some observations [23:32:16] Sun Grid Engine execd on yarrow is WARNING: NRPE: Unable to read output [23:43:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:43:55] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 194038.000000 [23:45:55] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:46:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:47:15] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63567.000000 [23:48:25] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [23:49:35] Cluster on nightshade is UNKNOWN: NRPE: Unable to read output [23:49:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:50:06] SMF on nightshade is UNKNOWN: NRPE: Unable to read output [23:54:45] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [23:57:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:59:16] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 556744 MB (10% inode=47%):