[00:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:11:23] Load avg. on willow is WARNING: WARNING - load average: 16.39, 15.33, 13.01 [00:12:23] Load avg. on willow is OK: OK - load average: 14.14, 14.81, 12.97 [00:16:43] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 58051 MB (5% inode=99%): [00:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:34:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1455258 [00:36:23] Load avg. on willow is WARNING: WARNING - load average: 13.23, 15.19, 14.63 [00:37:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:49:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1455845.000000 [00:57:22] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.74, 21.86, 20.13 [01:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:04:22] Load avg. on willow is WARNING: WARNING - load average: 15.77, 19.51, 19.95 [01:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:34:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1457762 [01:37:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:38:43] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 45474 MB (11% inode=99%): [01:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:45:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39834 MB (9% inode=99%): [01:49:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1458388.000000 [01:53:12] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:53:42] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 91758 MB (9% inode=98%): [01:55:12] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:04:22] Load avg. on willow is WARNING: WARNING - load average: 14.79, 16.37, 16.35 [02:14:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 91638 MB (9% inode=98%): [02:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:34:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1460091 [02:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:49:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1460568.000000 [03:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:04:23] Load avg. on willow is WARNING: WARNING - load average: 14.90, 16.90, 16.52 [03:10:53] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:34:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1462238 [03:36:22] Load avg. on willow is OK: OK - load average: 11.50, 13.53, 14.89 [03:37:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:40:33] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:49:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1462857.000000 [04:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:04:42] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24054 MB (5% inode=99%): [04:16:23] Load avg. on willow is WARNING: WARNING - load average: 14.63, 15.73, 15.02 [04:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:28:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44294 MB (10% inode=99%): [04:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:34:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1464612 [04:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:43:23] Load avg. on willow is OK: OK - load average: 11.18, 13.36, 14.84 [04:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:49:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1465241.000000 [05:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:16:23] Load avg. on willow is WARNING: WARNING - load average: 18.41, 17.55, 16.08 [05:19:23] @replag [05:19:23] jeremyb: s1-user: 2w 2d 23h 18m 5s [+0.66 s/s]; s3-rr-a: 20s [-0.00 s/s]; s3-user: 20s [-0.00 s/s] [05:27:23] Load avg. on willow is OK: OK - load average: 11.46, 14.00, 14.93 [05:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:34:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1466783 [05:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:45:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 27278 MB (2% inode=97%): [05:49:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1467344.000000 [05:54:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:54:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:54:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:55:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 91862 MB (9% inode=98%): [05:55:03] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39693 MB (9% inode=99%): [05:55:03] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 91862 MB (9% inode=98%): [06:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:06:23] Load avg. on willow is WARNING: WARNING - load average: 15.44, 15.79, 14.26 [06:08:23] Load avg. on willow is OK: OK - load average: 11.99, 14.44, 13.94 [06:15:23] Load avg. on willow is WARNING: WARNING - load average: 18.53, 16.82, 15.45 [06:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:30:23] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.87, 23.34, 19.71 [06:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:34:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1468988 [06:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:49:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1469504.000000 [07:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:11:22] Load avg. on willow is WARNING: WARNING - load average: 17.82, 16.42, 14.98 [07:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:29:22] Load avg. on willow is OK: OK - load average: 11.89, 13.34, 14.83 [07:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:34:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1470953 [07:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:49:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1471516.000000 [08:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:34:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1472607 [08:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:49:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1472891.000000 [09:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:34:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1473481 [09:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:49:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1473741.000000 [10:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:34:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1475235 [10:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:50:23] s1 replag on thyme is CRITICAL: (Service Check Timed Out) [11:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:25:53] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:34:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1477903 [11:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:40:53] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:51:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1478480.000000 [12:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:16:43] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 60477 MB (6% inode=99%): [12:21:05] Russell Blau * Re: [Toolserver-l] s1 replag update, suggestion, and question [12:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1479515 [12:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:37:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:51:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1480202.000000 [13:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:10:42] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24393 MB (5% inode=99%): [13:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:32:42] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 24811 MB (6% inode=99%): [13:33:43] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24263 MB (5% inode=99%): [13:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1481251 [13:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:51:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1481764.000000 [14:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:10:43] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 48617 MB (11% inode=99%): [14:17:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39088 MB (9% inode=99%): [14:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1482566 [14:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:39:04] DaB. * Re: [Toolserver-l] s1 replag update, suggestion, and question [14:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:51:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1482482.000000 [15:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1482657 [15:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:52:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1482532.000000 [16:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1481947 [16:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:52:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1481736.000000 [17:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:06:38] Hi all [17:08:19] Hi [17:08:51] @replag [17:08:52] se4598: s1-user: 2w 3d 3h 33m 27s [-0.16 s/s]; s3-rr-a: 45s [+0.00 s/s]; s3-user: 45s [+0.00 s/s] [17:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1481586 [17:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:45:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 30493 MB (2% inode=97%): [17:52:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1481375.000000 [17:55:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 92304 MB (9% inode=98%): [17:55:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 92304 MB (9% inode=98%): [18:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1481697 [18:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:52:33] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1481468.000000 [19:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:28:39] @replag [19:28:39] russblau: s1-user: 2w 3d 3h 29m 21s [-0.03 s/s]; s3-rr-a: 16s [-0.00 s/s]; s3-user: 16s [-0.00 s/s] [19:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1481298 [19:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:37:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:44:04] Carl (CBM) * Re: [Toolserver-l] s1 replag update, suggestion, and question [19:52:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1480779.000000 [19:53:02] @replag [19:53:03] russblau: s1-user: 2w 3d 3h 19m 12s [-0.42 s/s]; s3-rr-a: 35s [+0.01 s/s]; s3-user: 36s [+0.01 s/s]; s6-rr-a: 30s [+0.00 s/s]; s6-user: 30s [+0.00 s/s] [19:59:29] @replag [19:59:29] jeremyb: s1-user: 2w 3d 3h 16m 5s [-0.48 s/s]; s3-rr-a: 43s [+0.02 s/s]; s3-user: 43s [+0.02 s/s]; s5-rr-a: 10s [+0.00 s/s]; s5-user: 10s [+0.00 s/s] [20:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:00:22] 4.76 weeks at this rate [20:08:16] at least it's finally going in the right direction, but _so_ slowly [20:27:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1479586 [20:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:45:03] Load avg. on adenia is WARNING: WARNING - load average: 15.30, 14.44, 12.71 [20:49:02] Load avg. on adenia is OK: OK - load average: 14.25, 14.93, 13.36 [20:52:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1478470.000000 [20:57:03] Load avg. on adenia is WARNING: WARNING - load average: 15.41, 15.51, 14.30 [21:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1476290 [21:35:38] @replag [21:35:39] jeremyb: s1-user: 2w 3d 2h 4m 20s [-0.75 s/s]; s2-user: 39s [-]; s3-rr-a: 30s [-0.00 s/s]; s3-user: 30s [-0.00 s/s] [21:35:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:35:44] woot, better [21:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:52:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1475174.000000 [21:57:03] Load avg. on adenia is WARNING: WARNING - load average: 16.47, 16.04, 15.82 [22:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1472870 [22:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:45:03] Load avg. on adenia is OK: OK - load average: 12.20, 14.02, 14.82 [22:52:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1471873.000000 [23:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:12:03] Load avg. on adenia is WARNING: WARNING - load average: 15.80, 15.49, 14.84 [23:22:03] Load avg. on adenia is OK: OK - load average: 12.19, 14.36, 14.85 [23:27:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:32:02] Load avg. on adenia is WARNING: WARNING - load average: 16.77, 15.73, 15.17 [23:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1468982 [23:35:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:37:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:52:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1467771.000000