[00:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 133428 [00:01:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51066 [00:09:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133533.000000 [00:23:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:27:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:31:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:36:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51380.000000 [00:43:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:44:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134079 [01:01:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51641 [01:09:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134149.000000 [01:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:27:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:31:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:36:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51389.000000 [01:43:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:44:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 133916 [02:02:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51165 [02:09:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133965.000000 [02:12:39] /sql on thyme is WARNING: DISK WARNING - free space: /sql 173362 MB (18% inode=99%): [02:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:27:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:31:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:37:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50912.000000 [02:43:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:45:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134205 [03:02:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50815 [03:09:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134265.000000 [03:14:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 581305 MB (10% inode=48%): [03:18:10] /aux0 on hemlock is OK: DISK OK - free space: /aux0 581692 MB (11% inode=48%): [03:23:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:27:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:31:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:32:41] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:37:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50558.000000 [03:43:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:45:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:47:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106523 MB (10% inode=98%): [03:48:30] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106453 MB (10% inode=98%): [03:55:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:55:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 109270 MB (11% inode=98%): [03:56:30] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 108388 MB (11% inode=98%): [04:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134907 [04:02:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50855 [04:09:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 135078.000000 [04:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:28:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:32:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:33:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:38:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51159.000000 [04:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:46:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 135685 [05:02:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51580 [05:09:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 135795.000000 [05:14:30] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41780 MB (10% inode=99%): [05:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:24:50] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 56291 MB (4% inode=98%): [05:29:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:30:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23212 MB (5% inode=99%): [05:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:33:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:39:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51842.000000 [05:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:47:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:54:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 33450 MB (8% inode=99%): [05:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 136134 [06:03:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51832 [06:08:30] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24461 MB (5% inode=99%): [06:09:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 136142.000000 [06:15:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 61939 MB (15% inode=99%): [06:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:29:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:33:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:39:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51746.000000 [06:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:48:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:00:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 136517 [07:04:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51758 [07:09:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 136460.000000 [07:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:29:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:33:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:39:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51805.000000 [07:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:48:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 581547 MB (10% inode=48%): [08:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 136232 [08:04:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 51989 [08:09:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 136320.000000 [08:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:30:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:33:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:39:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52050.000000 [08:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:48:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 581398 MB (10% inode=48%): [09:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 136684 [09:04:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52090 [09:09:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 136758.000000 [09:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:30:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:34:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:39:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52213.000000 [09:44:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:49:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579712 MB (10% inode=48%): [10:00:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 137622 [10:04:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52350 [10:09:40] Load avg. on willow is WARNING: WARNING - load average: 22.84, 18.55, 15.39 [10:10:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.28, 21.18, 16.52 [10:10:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 137824.000000 [10:11:39] Load avg. on willow is WARNING: WARNING - load average: 19.62, 19.79, 16.32 [10:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:30:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:35:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:39:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52286.000000 [10:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:50:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:55:36] @replag [10:55:36] Merlissimo: s1-rr-a: 14h 31m 14s [+0.03 s/s]; s1-user: 1d 14h 21m 29s [+0.11 s/s]; s3-rr-a: 48s [+0.00 s/s]; s3-user: 48s [+0.00 s/s]; s7-rr-a: 11s [-0.00 s/s]; s7-user: 11s [-0.00 s/s] [10:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580921 MB (10% inode=48%): [11:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 138063 [11:04:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52364 [11:09:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39182 MB (9% inode=99%): [11:11:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138224.000000 [11:19:29] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24107 MB (5% inode=99%): [11:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:30:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:35:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 46919 MB (11% inode=99%): [11:35:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:39:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52511.000000 [11:42:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 32785 MB (8% inode=99%): [11:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:50:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:54:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 56819 MB (14% inode=99%): [11:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580697 MB (10% inode=48%): [12:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 138678 [12:04:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52981 [12:12:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138913.000000 [12:24:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:30:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:36:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:39:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 53473.000000 [12:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:50:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580584 MB (10% inode=48%): [13:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 139544 [13:04:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 53899 [13:12:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 139658.000000 [13:24:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:31:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:37:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:39:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54313.000000 [13:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:50:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580463 MB (10% inode=48%): [14:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 139749 [14:04:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54547 [14:12:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 139866.000000 [14:12:39] /sql on thyme is WARNING: DISK WARNING - free space: /sql 176733 MB (18% inode=99%): [14:24:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:31:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:37:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:39:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54629.000000 [14:44:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:51:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:57:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580297 MB (10% inode=48%): [15:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 140366 [15:05:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55184 [15:12:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 140411.000000 [15:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:31:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:37:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:39:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55511.000000 [15:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:51:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 580109 MB (10% inode=48%): [16:00:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 140733 [16:05:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55684 [16:12:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 140863.000000 [16:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:31:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:37:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:39:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55510.000000 [16:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:51:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579894 MB (10% inode=48%): [17:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 141120 [17:05:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55640 [17:08:15] @replag [17:08:15] Dispenser: s1-rr-a: 15h 27m 15s [+0.15 s/s]; s1-user: 1d 15h 14m 29s [+0.14 s/s]; s3-rr-a: 52s [+0.00 s/s]; s3-user: 52s [+0.00 s/s] [17:12:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 141313.000000 [17:22:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41848 MB (10% inode=99%): [17:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:24:50] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 59123 MB (4% inode=98%): [17:31:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:37:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:40:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55514.000000 [17:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:51:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 60113 MB (14% inode=99%): [17:51:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579725 MB (10% inode=48%): [18:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 141982 [18:06:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55635 [18:12:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 142126.000000 [18:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:32:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:36:30] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106216 MB (10% inode=98%): [18:36:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106212 MB (10% inode=98%): [18:37:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:40:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55440.000000 [18:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:49:29] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 108540 MB (11% inode=98%): [18:49:39] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 108683 MB (11% inode=98%): [18:51:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579551 MB (10% inode=48%): [19:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 142306 [19:06:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55408 [19:13:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 142439.000000 [19:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:32:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:37:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:40:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55214.000000 [19:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:52:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579669 MB (10% inode=48%): [20:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 143107 [20:06:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55331 [20:13:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 143373.000000 [20:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:32:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:38:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55257.000000 [20:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:52:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579492 MB (10% inode=48%): [21:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 144021 [21:06:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55249 [21:13:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 144270.000000 [21:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:33:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:38:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:41:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54887.000000 [21:44:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:50:04] nacht ts [21:52:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579313 MB (10% inode=48%): [22:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 144742 [22:06:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54732 [22:14:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 145087.000000 [22:24:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:33:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:33:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:38:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54384.000000 [22:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:52:30] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40449 MB (9% inode=99%): [22:53:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:56:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 579223 MB (10% inode=48%): [23:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 145562 [23:06:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54310 [23:14:39] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 145818.000000 [23:17:29] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 55419 MB (13% inode=99%): [23:23:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:33:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:33:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:38:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:40:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54167.000000 [23:44:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:53:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:56:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:57:09] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 576654 MB (10% inode=48%): [23:58:34] @replag [23:58:35] Ryan_Vesey: s1-rr-a: 15h 2m 39s [-0.06 s/s]; s1-user: 1d 16h 38m 30s [+0.20 s/s]; s2-user: 17s [+0.00 s/s]; s3-rr-a: 3m 39s [+0.01 s/s]; s3-user: 3m 39s [+0.01 s/s]; s5-rr-a: 16s [-]; s5-user: 16s [-] [23:59:05] Does anyone have the knowledge to explain the above to me? [23:59:34] which replag is the real one [23:59:40] er, the one I should be concerned with