[00:07:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109581.000000 [00:15:24] How do I ssh into willow from nightshade, `ssh willow` suspects a man in the middle attack. [00:15:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:21:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:28:20] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45714.000000 [00:29:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:30:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:37:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:51:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45871 [00:51:43] nacht ts [00:52:04] DaB. * Re: [Toolserver-announce] [Toolserver-l] Update of SGE on Thursday 5. July [00:52:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110503 [00:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:07:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110609.000000 [01:16:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:21:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:28:20] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46053.000000 [01:29:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:30:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:37:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:51:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46355 [01:53:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111048 [01:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:07:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 111243.000000 [02:12:40] /sql on thyme is WARNING: DISK WARNING - free space: /sql 176817 MB (18% inode=99%): [02:16:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:21:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:28:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46489.000000 [02:29:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:30:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:38:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:52:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46357 [02:53:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111842 [02:55:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:07:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112103.000000 [03:16:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:21:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:28:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46591.000000 [03:29:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:30:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:38:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:40:19] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106199 MB (10% inode=98%): [03:40:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106243 MB (10% inode=98%): [03:52:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46815 [03:53:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113178 [03:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:57:20] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 107182 MB (11% inode=98%): [03:57:39] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 107156 MB (11% inode=98%): [04:07:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113675.000000 [04:16:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:21:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:28:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46805.000000 [04:29:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:31:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:38:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:52:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 47311 [04:54:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 114395 [04:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:07:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114648.000000 [05:16:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:21:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:22:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:23:49] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 61525 MB (5% inode=98%): [05:28:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47985.000000 [05:29:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:32:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:38:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:52:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48276 [05:54:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 115407 [05:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:06:40] Load avg. on willow is WARNING: WARNING - load average: 21.42, 17.94, 14.77 [06:07:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 115662.000000 [06:16:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:20:40] Load avg. on willow is CRITICAL: CRITICAL - load average: 33.40, 22.27, 18.10 [06:21:39] Load avg. on willow is WARNING: WARNING - load average: 26.56, 22.29, 18.38 [06:21:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:28:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48319.000000 [06:29:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:32:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:38:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:45:39] Load avg. on willow is WARNING: WARNING - load average: 19.02, 15.81, 16.13 [06:51:39] Load avg. on willow is OK: OK - load average: 10.04, 13.09, 14.74 [06:53:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48383 [06:54:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 116365 [06:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:06:39] Load avg. on willow is WARNING: WARNING - load average: 12.64, 17.64, 16.37 [07:07:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116514.000000 [07:17:39] Load avg. on willow is OK: OK - load average: 13.04, 13.87, 14.95 [07:17:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:21:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:28:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48422.000000 [07:29:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:32:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:37:04] Bryan Tong Minh * Re: [Toolserver-l] Update of SGE on Thursday 5. July [07:38:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:53:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48405 [07:54:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 116863 [07:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:06:39] Load avg. on willow is WARNING: WARNING - load average: 12.96, 15.16, 13.25 [08:07:39] Load avg. on willow is OK: OK - load average: 12.66, 14.61, 13.18 [08:07:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117077.000000 [08:16:39] Load avg. on willow is WARNING: WARNING - load average: 17.75, 17.62, 15.29 [08:18:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:21:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:28:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48401.000000 [08:29:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:32:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:38:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:48:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106921 MB (10% inode=98%): [08:49:19] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106919 MB (10% inode=98%): [08:53:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48695 [08:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:55:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 117664 [09:06:19] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41511 MB (10% inode=99%): [09:07:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117957.000000 [09:12:40] Load avg. on willow is WARNING: WARNING - load average: 14.56, 16.09, 14.44 [09:13:39] Load avg. on willow is OK: OK - load average: 10.84, 14.75, 14.07 [09:18:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:21:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:28:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48578.000000 [09:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:31:19] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58081 MB (14% inode=99%): [09:32:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:38:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:53:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48254 [09:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:55:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119055 [10:08:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119255.000000 [10:12:09] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:12:49] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:14:29] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106781 MB (10% inode=98%): [10:14:39] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106781 MB (10% inode=98%): [10:18:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:21:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:21:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:28:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47418.000000 [10:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:32:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:38:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:40:09] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:50] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106755 MB (10% inode=98%): [10:53:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46682 [10:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:55:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119670 [11:08:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119804.000000 [11:13:45] hello all [11:17:04] DaB. * Re: [Toolserver-l] Update of SGE on Thursday 5. July [11:19:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:21:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:23:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:28:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45811.000000 [11:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:32:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:38:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:52:04] Bryan Tong Minh * Re: [Toolserver-l] Update of SGE on Thursday 5. July [11:53:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44818 [11:53:50] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:54:09] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:55:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:55:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 119976 [11:56:09] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106640 MB (10% inode=98%): [11:56:19] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106632 MB (10% inode=98%): [12:06:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 191.79, 109.56, 50.54 [12:08:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120182.000000 [12:20:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:21:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:28:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44236.000000 [12:30:08] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:32:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:35:40] Load avg. on willow is WARNING: WARNING - load average: 5.83, 7.26, 19.15 [12:38:19] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:40:09] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:40:59] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 106005 MB (10% inode=98%): [12:41:39] Load avg. on willow is OK: OK - load average: 6.45, 6.42, 14.82 [12:53:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43742 [12:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:55:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120823 [13:08:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121018.000000 [13:20:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:21:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:28:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42543.000000 [13:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:32:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:38:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:53:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42188 [13:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:55:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121448 [14:08:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121602.000000 [14:12:40] /sql on thyme is WARNING: DISK WARNING - free space: /sql 176455 MB (18% inode=99%): [14:17:59] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:20:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:22:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:29:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41001.000000 [14:30:10] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:32:29] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:32:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:38:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:53:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40484 [14:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:55:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 122000 [15:08:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122125.000000 [15:20:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:22:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:29:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40079.000000 [15:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:33:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:38:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:54:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40001 [15:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:55:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 122431 [16:08:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122607.000000 [16:20:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:21:34] @replag [16:21:35] Betacommand: s1-rr-a: 11h 7m 55s [-0.09 s/s]; s1-user: 1d 10h 6m 17s [+0.22 s/s]; s2-user: 14s [-0.00 s/s] [16:22:01] DaBPunkt: any idea why the replag on s1 is so high? [16:22:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:23:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:26:05] Betacommand: rev_sha1-population [16:29:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40110.000000 [16:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:34:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:39:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:53:50] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:54:20] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 106152 MB (10% inode=98%): [16:54:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40394 [16:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:55:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123029 [17:08:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123126.000000 [17:12:20] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40377 MB (9% inode=99%): [17:21:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:23:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:23:49] /sql on cassia is CRITICAL: DISK CRITICAL - free space: /sql 63013 MB (5% inode=99%): [17:24:59] Betacommand around? [17:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:30:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40638.000000 [17:31:16] ToAruShiroiNeko: yes [17:31:53] yay [17:32:06] could I interest you in a tool? :) [17:32:15] what? [17:32:45] I am hoping to get a tool developer to develop a tool to scan for double redirects [17:32:45] as in what special double redirect pages report [17:33:07] this is to identify double redirects that bots cant fix [17:34:11] ToAruShiroiNeko: run the bot and what ever is left is what cannot be fixed [17:34:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:34:52] yes [17:35:12] I wish there be a tool to list that in perhaps a table [17:35:20] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 56653 MB (13% inode=99%): [17:35:25] like how many entries are left unfixed per wiki [17:35:36] that could be clicked to go to the special page [17:35:44] a tool or one time query [17:37:40] @replag [17:37:40] Earwig: s1-rr-a: 11h 16m 41s [+0.12 s/s]; s1-user: 1d 10h 13m 1s [+0.09 s/s]; s2-user: 14s [-]; s3-rr-a: 1m 24s [+0.00 s/s]; s3-user: 1m 24s [+0.00 s/s] [17:39:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:40:48] ToAruShiroiNeko: what project would you like me to start with? [17:53:29] well [17:53:33] wikipedias perhaps [17:54:12] if a wiki has 0 unfixed double redirects its not interesting to me but if it has 1 or more then I want to know [17:54:23] if you can classify the double rediretcs that would be great [17:54:33] ie self redirects, redirect loops, protected pages [17:54:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 40712 [17:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:55:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123308 [18:02:37] ToAruShiroiNeko: you decided grepping your log file is too difficult? [18:03:15] valhallasw it is [18:03:20] because I dont know where to start [18:04:13] also going through all wikis takes me over several hours [18:04:16] ToAruShiroiNeko: by identifying which lines are relevant to you and finding out what the common thing in those lines is...? [18:04:39] also I wish it published collectively, ie statistics [18:06:48] grep logfile -e 'common thing' | wc -l ;-) [18:07:26] essentially - you very often request tools for very specific tasks. Learning some data analysis skills can be really helpful for that. [18:07:35] also in terms of career, if you care about that [18:08:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123597.000000 [18:13:09] * DarkoNeko chuckles [18:18:27] career? [18:18:44] Ah yes. A career in double redirects <3 [18:20:09] Professional double redirect exterminator. I need to soo get a ghost busters-like emergency vihecle [18:21:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:23:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:23:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:29:23] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 106983 MB (11% inode=98%): [18:30:08] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:30:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41036.000000 [18:34:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:36:19] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 105997 MB (10% inode=98%): [18:39:29] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:39:59] NTP on yarrow is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.105652 secs [18:47:59] NTP on yarrow is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [18:54:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41228 [18:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:55:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 124216 [19:08:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 124478.000000 [19:19:31] ToAruShiroiNeko: i already have that tool [19:20:48] it is used by xqbot every day for all wikis [19:22:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:23:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:30:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41565.000000 [19:34:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:40:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:52:59] Merlissimo link? [19:53:17] http://toolserver.org/~merl/redirects/double.php?dbname=dewiki [19:53:39] but i protect it for use by other bots [19:53:50] proect? [19:54:22] oh [19:54:25] thats not what I am seeking [19:54:32] I am seeking a list of issues bots cant fix [19:54:38] those were all fixed already [19:54:46] http://toolserver.org/~merl/redirects/broken.php?dbname=dewiki [19:54:58] no I mean double redirects on protected pages [19:54:59] NTP on yarrow is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset -0.100636 secs [19:55:00] self redirects [19:55:02] redirect loops [19:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:55:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 125075 [19:55:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41642 [19:55:46] ToAruShiroiNeko: for all wikis? [20:03:59] NTP on yarrow is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:08:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 125183.000000 [20:22:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:23:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:31:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42072.000000 [20:35:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:40:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:46:36] Merlissimo yes [20:46:54] well all wikis with a problem [20:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:55:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 125844 [20:55:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42281 [21:05:44] ToAruShiroiNeko: what want you do with the result? input for a program? [21:08:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 125996.000000 [21:22:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:23:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:31:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42696.000000 [21:35:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:40:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:55:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 126553 [21:55:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42855 [22:08:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126649.000000 [22:18:19] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41480 MB (10% inode=99%): [22:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:23:39] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:23:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:32:39] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42677.000000 [22:36:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:40:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:54:19] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 58891 MB (14% inode=99%): [22:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:55:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 127043 [22:55:39] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 42842 [23:08:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127174.000000 [23:22:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:23:40] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:23:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:30:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:32:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43120.000000 [23:36:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:41:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:55:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:55:19] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 127795 [23:56:40] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43477