[00:03:19] nacht ts [00:08:20] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127944.000000 [00:08:40] Load avg. on willow is WARNING: WARNING - load average: 15.41, 15.84, 12.37 [00:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:23:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:23:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:28:39] Load avg. on willow is OK: OK - load average: 11.42, 13.52, 14.75 [00:29:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107016 MB (11% inode=98%): [00:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:32:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43927.000000 [00:36:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:36:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106350 MB (10% inode=98%): [00:41:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:50:49] Merlissimo I want to do two things [00:50:56] one to let the local community know [00:51:01] so that they can fix them [00:51:23] two hopefully get automaiton in notifying the communities [00:51:44] redirect.py is ideal for this but I cannot convince people at the severity of the problem [00:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:55:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 128728 [00:56:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44136 [01:06:39] Load avg. on willow is WARNING: WARNING - load average: 17.02, 17.61, 15.71 [01:08:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 128882.000000 [01:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:23:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:23:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:24:39] Load avg. on willow is OK: OK - load average: 10.68, 12.76, 14.99 [01:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:32:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44524.000000 [01:36:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:41:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:55:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 129219 [01:56:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44858 [02:08:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 129289.000000 [02:10:40] Load avg. on willow is WARNING: WARNING - load average: 23.92, 19.43, 16.10 [02:12:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.27, 22.63, 17.67 [02:12:39] /sql on thyme is WARNING: DISK WARNING - free space: /sql 174861 MB (18% inode=99%): [02:13:40] Load avg. on willow is WARNING: WARNING - load average: 24.29, 22.41, 17.92 [02:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:23:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:23:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:30:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.34, 20.11, 18.20 [02:32:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45126.000000 [02:36:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:41:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:56:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 129586 [02:56:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45618 [03:06:39] Load avg. on willow is WARNING: WARNING - load average: 20.98, 19.57, 16.54 [03:08:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 129788.000000 [03:14:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.13, 24.39, 20.05 [03:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:23:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:24:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:30:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:33:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46075.000000 [03:36:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:42:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:48:39] Load avg. on willow is WARNING: WARNING - load average: 15.56, 17.55, 19.93 [03:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:56:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 130609 [03:57:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46061 [04:08:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130860.000000 [04:11:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.69, 22.82, 20.07 [04:23:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:24:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:24:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:28:40] Load avg. on willow is WARNING: WARNING - load average: 15.11, 18.38, 19.84 [04:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:33:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45942.000000 [04:36:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:42:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:56:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 131858 [04:57:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46338 [04:59:40] Load avg. on willow is OK: OK - load average: 10.51, 11.54, 14.86 [05:03:20] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107248 MB (11% inode=98%): [05:03:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107093 MB (11% inode=98%): [05:08:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 131998.000000 [05:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:24:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:24:50] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 58462 MB (4% inode=98%): [05:25:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:34:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46581.000000 [05:37:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:42:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:43:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 105763 MB (10% inode=98%): [05:44:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107038 MB (11% inode=98%): [05:51:39] Load avg. on willow is WARNING: WARNING - load average: 11.16, 21.32, 18.07 [05:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:55:19] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106723 MB (10% inode=98%): [05:55:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106724 MB (10% inode=98%): [05:57:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 132660 [05:57:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46739 [06:00:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 34.87, 21.94, 18.86 [06:01:39] Load avg. on willow is WARNING: WARNING - load average: 27.17, 22.01, 19.08 [06:08:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 132858.000000 [06:10:19] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107047 MB (11% inode=98%): [06:10:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107047 MB (11% inode=98%): [06:23:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:24:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:25:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:27:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106605 MB (10% inode=98%): [06:28:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106142 MB (10% inode=98%): [06:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:34:10] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46573.000000 [06:37:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:42:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:48:02] Things are blowing up more than usual. [06:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:55:13] things have been really blown-uppy recently [06:55:14] :( [06:55:28] replag and WMF-side maxlag are way too high [06:57:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 133547 [06:57:20] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 42887 MB (10% inode=99%): [06:57:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46667 [07:08:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133724.000000 [07:12:39] Load avg. on willow is WARNING: WARNING - load average: 29.60, 21.75, 15.99 [07:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:23:39] Load avg. on willow is OK: OK - load average: 10.00, 13.69, 14.62 [07:24:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:25:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:28:20] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 59404 MB (14% inode=99%): [07:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:34:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46869.000000 [07:37:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:42:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:50:40] Load avg. on willow is WARNING: WARNING - load average: 14.82, 19.29, 16.78 [07:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:55:39] Load avg. on willow is OK: OK - load average: 11.20, 13.51, 14.87 [07:58:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134051 [07:58:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47346 [08:06:49] Load avg. on willow is WARNING: WARNING - load average: 15.64, 17.40, 16.66 [08:08:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134411.000000 [08:18:40] Load avg. on willow is OK: OK - load average: 7.69, 11.95, 14.84 [08:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:24:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:25:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:27:20] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107688 MB (11% inode=98%): [08:27:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107702 MB (11% inode=98%): [08:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:34:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47510.000000 [08:38:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:42:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:43:20] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106533 MB (10% inode=98%): [08:43:40] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106533 MB (10% inode=98%): [08:54:11] @replag [08:54:11] Dispenser: s1-rr-a: 13h 14m 57s [+0.13 s/s]; s1-user: 1d 13h 28m 33s [+0.21 s/s]; s3-rr-a: 44s [-0.00 s/s]; s3-user: 44s [-0.00 s/s] [08:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:58:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134889 [08:58:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47697 [09:08:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134974.000000 [09:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:24:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:25:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:34:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47577.000000 [09:38:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:42:20] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107095 MB (11% inode=98%): [09:42:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:49:30] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106222 MB (10% inode=98%): [09:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:58:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 135119 [09:58:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47455 [10:06:46] hi, wondered why there is such a large replag atm - http://en.wikipedia.org/wiki/User:JaGa/Short_leaderboard shows 37.5 hours [10:08:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 135163.000000 [10:10:41] @replag [10:10:41] Rcsprinter: s1-rr-a: 13h 10m 31s [-0.06 s/s]; s1-user: 1d 13h 32m 44s [+0.05 s/s]; s3-rr-a: 49s [+0.00 s/s]; s3-user: 49s [+0.00 s/s] [10:23:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:24:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:26:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:30:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:34:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47524.000000 [10:38:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:42:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:57:29] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 106962 MB (11% inode=98%): [10:57:39] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107316 MB (11% inode=98%): [10:58:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47793 [10:59:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134696 [11:04:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106653 MB (10% inode=98%): [11:04:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106612 MB (10% inode=98%): [11:08:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134634.000000 [11:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:24:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:26:40] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107470 MB (11% inode=98%): [11:26:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:27:29] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 108756 MB (11% inode=98%): [11:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:34:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47759.000000 [11:38:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:58:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47633 [11:59:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134657 [12:08:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134716.000000 [12:23:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:25:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:26:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:35:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47968.000000 [12:38:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:42:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:58:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48181 [12:59:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 134099 [13:08:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134052.000000 [13:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:26:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:27:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:35:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48290.000000 [13:39:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:55:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:58:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48320 [13:59:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 133441 [14:08:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133322.000000 [14:09:50] @replag [14:09:50] Dispenser: s1-rr-a: 13h 27m 2s [+0.07 s/s]; s1-user: 1d 13h 2m 2s [-0.13 s/s]; s3-rr-a: 58s [+0.00 s/s]; s3-user: 58s [+0.00 s/s] [14:12:39] /sql on thyme is WARNING: DISK WARNING - free space: /sql 174576 MB (18% inode=99%): [14:15:43] @replag [14:15:44] DarkoNeko: s1-rr-a: 13h 26m 53s [-0.03 s/s]; s1-user: 1d 13h 1m 23s [-0.11 s/s]; s3-rr-a: 15s [-0.12 s/s]; s3-user: 15s [-0.12 s/s] [14:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:26:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:27:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:35:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48443.000000 [14:40:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:55:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:56:24] @replag [14:56:24] Merlissimo: s1-rr-a: 13h 29m 59s [+0.09 s/s]; s1-user: 1d 12h 48m 5s [-0.33 s/s]; s3-rr-a: 3m 22s [+0.08 s/s]; s3-user: 3m 22s [+0.08 s/s]; s4-user: 1m 50s [-0.00 s/s]; s6-rr-a: 3m 38s [+0.09 s/s]; s6-user: 3m 38s [+0.09 s/s]; s7-rr-a: 3m 4s [-0.00 s/s] [14:56:25] Merlissimo: s7-user: 3m 4s [-0.00 s/s] [14:58:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48602 [14:59:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 132419 [15:08:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 132124.000000 [15:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:26:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:27:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:35:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48558.000000 [15:40:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:58:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48847 [15:59:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 130925 [16:09:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130807.000000 [16:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:26:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:28:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:35:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49118.000000 [16:41:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:59:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 130107 [16:59:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49197 [17:08:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130108.000000 [17:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:24:49] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 61323 MB (5% inode=98%): [17:26:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:28:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:35:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49262.000000 [17:41:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:59:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 130737 [17:59:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49372 [18:08:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130914.000000 [18:23:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:26:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:29:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:31:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:35:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49472.000000 [18:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:42:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:59:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 131536 [18:59:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49645 [19:08:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 131616.000000 [19:17:09] @replag [19:17:10] Dispenser: s1-rr-a: 13h 47m 48s [+0.07 s/s]; s1-user: 1d 12h 34m 49s [-0.05 s/s] [19:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:26:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:29:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:35:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 49615.000000 [19:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:43:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:55:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:59:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 132273 [20:00:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49925 [20:08:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 132377.000000 [20:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:26:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:29:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:35:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50114.000000 [20:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:43:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:55:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:00:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 132702 [21:00:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50318 [21:09:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 132791.000000 [21:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:26:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:29:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:35:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50486.000000 [21:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:43:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 133018 [22:00:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50538 [22:08:21] @replag [22:08:21] Earwig: s1-rr-a: 14h 2m 12s [+0.08 s/s]; s1-user: 1d 12h 59m 37s [+0.14 s/s]; s2-user: 13s [-0.00 s/s]; s3-rr-a: 55s [-0.01 s/s]; s3-user: 55s [-0.01 s/s] [22:09:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133183.000000 [22:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:27:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:30:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:31:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:32:05] @replag [22:32:05] Earwig: s1-rr-a: 14h 3m 17s [+0.05 s/s]; s1-user: 1d 13h 30s [+0.04 s/s]; s3-rr-a: 1m 38s [+0.03 s/s]; s3-user: 1m 38s [+0.03 s/s] [22:35:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50590.000000 [22:42:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:43:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:00:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 133383 [23:01:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50929 [23:09:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133513.000000 [23:11:25] * Dispenser wonders if we'll still have replag during Wikimania [23:15:29] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40293 MB (9% inode=99%): [23:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:26:00] @replag [23:26:00] Brooke: s1-rr-a: 14h 9m 14s [+0.11 s/s]; s1-user: 1d 13h 7m 14s [+0.12 s/s]; s3-rr-a: 15s [-0.03 s/s]; s3-user: 16s [-0.03 s/s] [23:26:10] It's not terrible. [23:26:15] I'd like to see thyme get fixed, though. [23:26:34] I wonder which is worse: having corrupt data or no data. [23:26:39] It could be taken out of rotation. [23:27:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:30:49] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:31:08] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:36:30] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50933.000000 [23:38:30] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 57940 MB (14% inode=99%): [23:43:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:43:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output