[00:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:16:43] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 95866 MB (9% inode=99%): [00:17:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [00:29:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1403609 [00:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:37:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [00:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:53:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1403525.000000 [01:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:16:23] Load avg. on willow is WARNING: WARNING - load average: 15.38, 15.98, 14.00 [01:17:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:18:23] Load avg. on willow is OK: OK - load average: 12.25, 14.62, 13.73 [01:30:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1402681 [01:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [01:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:53:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1402032.000000 [01:55:12] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:55:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:56:03] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89024 MB (9% inode=98%): [01:56:03] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38560 MB (9% inode=99%): [01:56:04] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89024 MB (9% inode=98%): [02:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:01:12] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:01:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:01:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:17:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [02:20:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89519 MB (9% inode=98%): [02:20:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89519 MB (9% inode=98%): [02:21:23] Load avg. on willow is WARNING: WARNING - load average: 18.98, 16.57, 14.25 [02:27:12] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89437 MB (9% inode=98%): [02:27:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89437 MB (9% inode=98%): [02:30:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1401469 [02:35:23] Load avg. on willow is OK: OK - load average: 12.85, 14.73, 14.89 [02:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [02:41:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:41:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:53:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1401058.000000 [03:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:00:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89333 MB (9% inode=98%): [03:00:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89333 MB (9% inode=98%): [03:00:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38548 MB (9% inode=99%): [03:17:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [03:25:05] sulinfo is running really slowly [03:25:12] (but apparently not totally broken [03:25:13] ) [03:25:22] damnit, http://xkcd.com/859/ [03:30:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1400835 [03:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [03:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:53:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1400945.000000 [03:53:57] @replag [03:53:59] jeremyb: s1-user: 2w 2d 5h 9m 5s [-0.21 s/s]; s2-user: 34s [+0.00 s/s]; s3-rr-a: 1m 6s [+0.00 s/s]; s3-user: 1m 7s [+0.00 s/s] [03:55:03] this is a pretty damn slow recovery! [04:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:17:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [04:29:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:29:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:29:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:30:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:31:22] Load avg. on willow is WARNING: WARNING - load average: 18.75, 17.48, 15.16 [04:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:34:13] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:34:13] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:34:13] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:34:23] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:34:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89218 MB (9% inode=98%): [04:34:42] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89219 MB (9% inode=98%): [04:34:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38777 MB (9% inode=99%): [04:34:52] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:35:02] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:35:03] SMTP on z-dat-s4-a is OK: SMTP OK - 0.007 sec. response time [04:35:03] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1400045 [04:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [04:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:53:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1399974.000000 [04:58:23] Load avg. on willow is OK: OK - load average: 12.34, 13.41, 14.85 [05:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:06:22] Load avg. on willow is WARNING: WARNING - load average: 14.38, 14.94, 15.30 [05:09:22] Load avg. on willow is OK: OK - load average: 12.39, 14.11, 14.92 [05:17:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [05:21:02] Load avg. on ortelius is WARNING: WARNING - load average: 13.45, 15.68, 13.47 [05:25:03] Load avg. on ortelius is OK: OK - load average: 13.53, 14.76, 13.62 [05:30:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1399423 [05:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:37:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:45:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 26015 MB (2% inode=97%): [05:54:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1399202.000000 [06:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:03:53] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 10.259 second response time [06:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1399073 [06:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:37:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:45:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [06:49:53] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 16.601 second response time [06:54:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1398557.000000 [07:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:10:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:26:53] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 18.303 second response time [07:28:43] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24233 MB (5% inode=99%): [07:31:02] fisheye.toolserver.org on web.amaranth is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 272 bytes in 20.465 second response time [07:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:53] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 272 bytes in 19.529 second response time [07:32:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1397463 [07:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:52:02] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [07:54:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1397178.000000 [08:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:15:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39358 MB (9% inode=99%): [08:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:32:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1396550 [08:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [08:53:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:53:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:53:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:53:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 90169 MB (9% inode=98%): [08:53:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 90169 MB (9% inode=98%): [08:53:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39344 MB (9% inode=99%): [08:54:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1396318.000000 [09:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:32:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1396154 [09:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:52:04] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [09:54:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1396016.000000 [10:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:32:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1394742 [10:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [10:54:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1393936.000000 [10:54:13] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:54:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:54:13] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:54:43] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 89666 MB (9% inode=98%): [10:54:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 89667 MB (9% inode=98%): [10:54:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39283 MB (9% inode=99%): [11:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:32:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1392294 [11:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [11:54:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1391380.000000 [12:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:16:43] /sql on thyme is CRITICAL: DISK CRITICAL - free space: /sql 96619 MB (9% inode=99%): [12:19:04] @replag [12:19:05] se4598: s1-user: 2w 2d 2h 17m 53s [-0.50 s/s]; s2-user: 18s [-0.00 s/s]; s3-rr-a: 28s [+0.00 s/s]; s3-user: 28s [+0.00 s/s] [12:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:32:32] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1390237 [12:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:38:22] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:41:04] Danny B. * [Toolserver-l] Cron on nightshade [12:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [12:54:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1389589.000000 [13:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:16:43] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 24373 MB (5% inode=99%): [13:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:32:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1388575 [13:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:42:43] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39210 MB (9% inode=99%): [13:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [13:54:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1387885.000000 [14:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:32:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1387090 [14:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [14:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [14:54:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1386469.000000 [15:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:31:24] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:32:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1385751 [15:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [15:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [15:54:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1385088.000000 [16:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:21:15] @replag [16:21:15] jeremyb: s1-user: 2w 2d 35m 33s [-0.42 s/s]; s2-user-c: 18s [-0.01 s/s]; s3-rr-a: 27s [-0.00 s/s]; s3-user: 27s [-0.00 s/s]; s5-user-c: 18s [-0.01 s/s] [16:21:27] jeopardy music anyone? [16:23:28] http://munin.toolserver.org/Database/thyme/mysql_replication.html [16:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:32:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1384596 [16:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [16:54:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1384349.000000 [17:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:32:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1383789 [17:35:41] thanks tsnag! [17:35:46] @replag [17:35:46] jeremyb: s1-user: 2w 2d 23m 2s [-0.17 s/s]; s3-rr-a: 29s [+0.00 s/s]; s3-user: 29s [+0.00 s/s] [17:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:45:43] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 24985 MB (2% inode=97%): [17:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [17:54:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1383452.000000 [18:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:33:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1382848 [18:35:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:37:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:50:59] hello all [18:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [18:54:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1382518.000000 [19:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:00:05] DaB. * Re: [Toolserver-l] Cron on nightshade [19:01:04] DaB. * Re: [Toolserver-l] http://status.toolserver.org/ Down [19:08:05] Mike Dupont * Re: [Toolserver-l] Cron on nightshade [19:31:22] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:31:32] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:33:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:34:03] Load avg. on ortelius is WARNING: WARNING - load average: 19.96, 18.21, 13.44 [19:35:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1381426 [19:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:36:22] @replag [19:36:24] jeremyb: s1-user: 2w 1d 23h 43m 9s [-0.33 s/s] [19:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:51:32] giftpflanze: "am 12. Aug. 2012, 21:22 geschützt, ‎[edit=autoconfirmed]" looks like the giftbot has umlaut-problems [19:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [19:54:02] Load avg. on ortelius is OK: OK - load average: 11.77, 14.00, 14.82 [19:54:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1380749.000000 [20:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:17:09] @replag [20:17:10] russblau: s1-rr-a: 1m 11s [+0.00 s/s]; s1-user: 2w 1d 23h 22m 44s [-0.50 s/s]; s2-user: 10s [-0.00 s/s]; s3-rr-a: 14s [-0.00 s/s]; s3-user: 14s [-0.00 s/s] [20:18:59] DaBPunkt: hello; I hate to keep harping on thyme but this replag has made my project's tools mostly useless, and another month or two of recovery will be extremely painful. Is it possible, as a test, to disable all user access to thyme (yes, breaking all tools, I know) for an hour or so, just to see what effect that has on the replag? [20:19:42] russblau: I try to announce to moving of the user-databases to rosemary – but my mail-client is hating me at the moment [20:20:01] ooooh [20:20:16] DaBPunkt: oh, that's great (well, not the part about your mail-client, I think you should hate it back) [20:24:21] https://jira.toolserver.org/browse/MNT-1262 [20:25:40] thank you [20:29:56] np [20:31:23] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:33:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1379925 [20:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:36:41] DaBPunkt: is there something i can help with? how's puppetizing going? how many active admins are there right now? [20:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:40:57] (and in general the abandoning of solaris) [20:43:36] the puppetizing is working quite well. 2 of 2 roots are active at the moment [20:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:44:44] DaBPunkt: 87.98.138.186 is making alot of queries to my backlinkscount tool, like 20 per second [20:45:05] Is that an abuse of Toolserver resources? [20:48:48] Do you think it is a abuse? Can there be a valid reason? [20:52:03] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [20:53:13] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:53:43] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 90228 MB (9% inode=98%): [20:54:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1379348.000000 [20:55:02] Probably trying to build a backlinks database [20:56:17] Far better done through SQL. I'm blocking requests without a user-agent now. [20:59:55] that's always a good idea :) [21:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:03:01] @replag [21:03:01] Brooke: s1-user: 2w 1d 23h 4m 35s [-0.40 s/s]; s2-user: 12s [+0.00 s/s]; s3-rr-a: 12s [-0.00 s/s]; s3-user: 12s [-0.00 s/s] [21:07:43] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 7.919 second response time [21:31:24] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:31:33] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:32:23] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:33:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:34:33] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1378241 [21:35:23] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:54:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1377773.000000 [22:00:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:33:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1376725 [22:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:43:53] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:47:20] nacht ts [22:54:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1376163.000000 [22:54:42] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 90042 MB (9% inode=98%): [23:00:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:26:03] SSH on damiana is CRITICAL: Server answer: [23:34:33] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:35:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1374874 [23:35:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:39:03] SSH on turnera is CRITICAL: Server answer: [23:43:03] SSH on turnera is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [23:43:52] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:50:03] SSH on turnera is CRITICAL: Server answer: [23:54:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1374466.000000