[00:00:18] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1219508.000000 [00:01:59] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [00:04:38] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8992.000000 [00:05:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:05:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:08:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:11:07] Load avg. on willow is WARNING: WARNING - load average: 16.31, 14.93, 13.21 [00:12:07] Load avg. on willow is OK: OK - load average: 12.77, 14.20, 13.06 [00:16:48] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:22:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:33:07] MySQL slave on thyme is WARNING: No slaves defined [00:33:53] 3(assigned) [UTRS-92] Appeal in backlog when closed <10https://jira.toolserver.org/browse/UTRS-92> (TParis) [00:33:54] 3(resolved) [UTRS-92] Appeal in backlog when closed <10https://jira.toolserver.org/browse/UTRS-92> (TParis) [00:35:47] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.379395/1.75, alarm hl:np_load_avg=1.439453/2.0, alarm hl:mem_free=204.000000M/350M, alarm hl:available=1/0 [00:35:54] 3(commented) [UTRS-87] Log timestamps are sometimes incorrect <10https://jira.toolserver.org/browse/UTRS-87> (TParis) [00:36:47] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:39:47] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.168457/1.75, alarm hl:np_load_avg=1.279785/2.0, alarm hl:mem_free=241.000000M/350M, alarm hl:available=1/0 [00:53:07] Load avg. on willow is WARNING: WARNING - load average: 15.59, 15.17, 13.53 [00:58:48] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:00:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1223114.000000 [01:02:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [01:04:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10492.000000 [01:06:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:06:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:08:17] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:09:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:17:48] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:22:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:33:08] MySQL slave on thyme is WARNING: No slaves defined [01:33:47] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:37:47] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.653320/1.75, alarm hl:np_load_avg=1.851074/2.0, alarm hl:mem_free=319.000000M/350M, alarm hl:available=1/0 [01:47:18] Load avg. on willow is OK: OK - load average: 11.23, 12.94, 14.80 [01:52:17] Load avg. on willow is WARNING: WARNING - load average: 17.88, 15.28, 15.19 [01:58:18] Load avg. on willow is OK: OK - load average: 10.32, 13.99, 14.88 [02:01:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1226774.000000 [02:02:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [02:04:48] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12323.000000 [02:06:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:06:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:09:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:17:48] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:20:17] Load avg. on willow is WARNING: WARNING - load average: 20.61, 17.77, 17.28 [02:22:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:33:06] MySQL slave on thyme is WARNING: No slaves defined [02:37:47] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.936524/1.75, alarm hl:np_load_avg=2.099609/2.0, alarm hl:mem_free=248.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.936524/1.9, alarm hl:np_load_long=2.031738/2.25, alarm hl:mem_free=248.000000M/200M, alarm hl:available=1/0 [03:01:29] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1230377.000000 [03:03:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [03:04:51] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15591.000000 [03:06:50] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:06:50] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:09:39] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:17:49] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:19:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:20:17] Load avg. on willow is WARNING: WARNING - load average: 13.92, 14.20, 15.19 [03:21:18] Load avg. on willow is OK: OK - load average: 11.54, 13.45, 14.86 [03:22:49] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:22:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.383789/1.75, alarm hl:np_load_avg=1.608887/2.0, alarm hl:mem_free=284.000000M/350M, alarm hl:available=1/0 [03:32:47] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:33:20] MySQL slave on thyme is WARNING: No slaves defined [03:42:49] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.310547/1.75, alarm hl:np_load_avg=1.409180/2.0, alarm hl:mem_free=305.000000M/350M, alarm hl:available=1/0 [03:48:40] [[System administrators]] ! 10https://wiki.toolserver.org/w/index.php?diff=6987&oldid=6776&rcid=9199 * 198.228.195.137 * (-3) (/* admins */ ) [03:49:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:54:29] [[System administrators]] 10https://wiki.toolserver.org/w/index.php?diff=6988&oldid=6987&rcid=9200 * MZMcBride * (+3) (rvv) [04:01:38] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1233988.000000 [04:03:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [04:04:49] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18820.000000 [04:06:39] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 114660 MB (11% inode=99%): [04:07:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:07:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:09:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:11:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.033691/1.75, alarm hl:np_load_avg=1.750976/2.0, alarm hl:mem_free=153.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.033691/1.9, alarm hl:np_load_long=1.625976/2.25, alarm hl:mem_free=153.000000M/200M, alarm hl:available=1/0 [04:12:17] Load avg. on willow is WARNING: WARNING - load average: 16.22, 14.25, 13.14 [04:14:17] Load avg. on willow is OK: OK - load average: 14.43, 14.11, 13.22 [04:17:49] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.083008/1.00, alarm hl:np_load_long=0.651367/1.50, alarm hl:mem_free=18655.000000M/350M, alarm hl:available=1/0 [04:18:07] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:18:48] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:22:48] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:23:48] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.314453/1.10, alarm hl:np_load_long=0.782227/1.55, alarm hl:mem_free=18968.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.314453/1.00, alarm hl:np_load_long=0.782227/1.50, alarm hl:mem_free=18968.000000M/350M, alarm hl:available=1/0 [04:29:47] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:34:18] MySQL slave on thyme is WARNING: No slaves defined [04:40:48] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:41:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.348633/1.75, alarm hl:np_load_avg=1.353516/2.0, alarm hl:mem_free=268.000000M/350M, alarm hl:available=1/0 [04:43:48] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.646484/1.10, alarm hl:np_load_long=1.238281/1.55, alarm hl:mem_free=18843.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.646484/1.00, alarm hl:np_load_long=1.238281/1.50, alarm hl:mem_free=18843.000000M/350M, alarm hl:available=1/0 [04:47:48] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:47:57] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:01:40] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1237588.000000 [05:03:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [05:05:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21814.000000 [05:07:57] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:07:57] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:09:59] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:15:40] Load avg. on willow is WARNING: WARNING - load average: 18.38, 15.53, 13.58 [05:18:08] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:19:40] Load avg. on willow is OK: OK - load average: 12.91, 14.84, 13.82 [05:22:58] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:26:57] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.278320/1.75, alarm hl:np_load_avg=1.871094/2.0, alarm hl:mem_free=136.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.278320/1.9, alarm hl:np_load_long=1.753906/2.25, alarm hl:mem_free=136.000000M/200M, alarm hl:available=1/0 [05:27:40] Load avg. on willow is WARNING: WARNING - load average: 17.40, 15.56, 14.32 [05:34:18] MySQL slave on thyme is WARNING: No slaves defined [05:37:41] Load avg. on willow is OK: OK - load average: 10.95, 14.83, 14.89 [06:01:49] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1241194.000000 [06:03:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [06:05:49] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24725.000000 [06:07:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:07:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:10:07] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:11:59] @replag [06:11:59] Joan: s1-rr-a: 2w 8h 56m 55s [+1.00 s/s]; s1-user: 2w 8h 56m 55s [+1.00 s/s]; s2-user: 37s [-0.00 s/s]; s2-user-c: 6h 56m 56s [+0.55 s/s]; s3-rr-a: 46s [+0.00 s/s]; s3-user: 46s [+0.00 s/s]; s5-user-c: 6h 56m 56s [+0.55 s/s]; s6-rr-a: 17s [+0.00 s/s] [06:12:00] Joan: s6-user: 17s [+0.00 s/s] [06:19:07] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:20:57] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:22:57] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:23:57] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.510742/1.10, alarm hl:np_load_long=1.001953/1.55, alarm hl:mem_free=19764.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.510742/1.00, alarm hl:np_load_long=1.001953/1.50, alarm hl:mem_free=19764.000000M/350M, alarm hl:available=1/0 [06:26:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.874512/1.75, alarm hl:np_load_avg=1.746094/2.0, alarm hl:mem_free=303.000000M/350M, alarm hl:available=1/0 [06:29:57] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:31:57] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [06:33:37] Load avg. on willow is WARNING: WARNING - load average: 10.66, 13.27, 15.24 [06:34:37] Load avg. on willow is OK: OK - load average: 10.32, 12.72, 14.92 [06:35:18] MySQL slave on thyme is WARNING: No slaves defined [06:35:58] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.956055/1.10, alarm hl:np_load_long=1.191406/1.55, alarm hl:mem_free=20183.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.956055/1.00, alarm hl:np_load_long=1.191406/1.50, alarm hl:mem_free=20183.000000M/350M, alarm hl:available=1/0 [06:45:47] Load avg. on willow is WARNING: WARNING - load average: 19.57, 16.24, 15.34 [06:53:48] Load avg. on willow is OK: OK - load average: 10.95, 14.27, 14.97 [06:57:00] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:01:47] Load avg. on willow is WARNING: WARNING - load average: 17.84, 17.23, 15.80 [07:02:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1244802.000000 [07:02:00] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.405274/1.10, alarm hl:np_load_long=1.471680/1.55, alarm hl:mem_free=19164.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.405274/1.00, alarm hl:np_load_long=1.471680/1.50, alarm hl:mem_free=19164.000000M/350M, alarm hl:available=1/0 [07:03:00] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:03:17] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [07:05:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26839.000000 [07:07:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:07:58] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:10:17] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:15:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.997559/1.75, alarm hl:np_load_avg=2.077637/2.0, alarm hl:mem_free=346.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.997559/1.9, alarm hl:np_load_long=2.050293/2.25, alarm hl:mem_free=346.000000M/200M, alarm hl:available=1/0 [07:17:58] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:19:07] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:23:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.997559/1.75, alarm hl:np_load_avg=1.990234/2.0, alarm hl:mem_free=491.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.997559/1.9, alarm hl:np_load_long=2.007324/2.25, alarm hl:mem_free=491.000000M/200M, alarm hl:available=1/0 [07:23:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:35:17] Load avg. on adenia is WARNING: WARNING - load average: 19.42, 13.49, 7.75 [07:35:59] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.082031/1.00, alarm hl:np_load_long=0.994140/1.50, alarm hl:mem_free=17878.000000M/350M, alarm hl:available=1/0 [07:36:19] MySQL slave on thyme is WARNING: No slaves defined [07:37:00] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:45:16] Load avg. on adenia is OK: OK - load average: 7.32, 13.96, 11.59 [07:57:17] Load avg. on adenia is WARNING: WARNING - load average: 18.93, 13.64, 11.30 [08:00:47] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.16, 19.89, 18.06 [08:01:48] Load avg. on willow is WARNING: WARNING - load average: 22.14, 19.29, 17.96 [08:02:18] Load avg. on adenia is OK: OK - load average: 11.84, 14.55, 12.55 [08:02:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1248463.000000 [08:03:18] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [08:05:09] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.102539/1.10, alarm hl:np_load_long=0.910156/1.55, alarm hl:mem_free=19840.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.102539/1.00, alarm hl:np_load_long=0.910156/1.50, alarm hl:mem_free=19840.000000M/350M, alarm hl:available=1/0 [08:06:58] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30083.000000 [08:08:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:08:07] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:09:07] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:10:26] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:17:08] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.401367/1.10, alarm hl:np_load_long=0.984375/1.55, alarm hl:mem_free=19813.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.401367/1.00, alarm hl:np_load_long=0.984375/1.50, alarm hl:mem_free=19813.000000M/350M, alarm hl:available=1/0 [08:19:09] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:20:07] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:23:09] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:23:09] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.595215/1.75, alarm hl:np_load_avg=1.948242/2.0, alarm hl:mem_free=329.000000M/350M, alarm hl:available=1/0 [08:28:58] Load avg. on willow is OK: OK - load average: 12.18, 13.35, 14.98 [08:31:23] Hi, I've been getting an error message "Toolserver servers are currently experiencing technical difficulties. ..." when visiting my http://toolserver.org/~luxo/gwatch/watchlist.php since yesterday. Can someone help, please ? [08:32:55] The message asks me to provide this line: "Request: GET from 91.198.174.204 to toolserver.org Error: Line 216 (eswiki_p) at Tue, 03 Apr 2012 08:26:02 +0000 UTC" [08:32:59] Load avg. on willow is WARNING: WARNING - load average: 14.69, 14.97, 15.35 [08:36:18] MySQL slave on thyme is WARNING: No slaves defined [08:39:58] Load avg. on willow is OK: OK - load average: 12.86, 13.96, 14.91 [08:44:08] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.486328/1.10, alarm hl:np_load_long=0.968750/1.55, alarm hl:mem_free=19083.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.486328/1.00, alarm hl:np_load_long=0.968750/1.50, alarm hl:mem_free=19083.000000M/350M, alarm hl:available=1/0 [08:46:08] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:52:08] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:58:08] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.574707/1.75, alarm hl:np_load_avg=1.539551/2.0, alarm hl:mem_free=324.000000M/350M, alarm hl:available=1/0 [09:03:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [09:03:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1252128.000000 [09:07:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33399.000000 [09:08:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:08:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:10:26] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:10:57] Load avg. on willow is WARNING: WARNING - load average: 19.39, 16.88, 15.18 [09:17:58] Load avg. on willow is OK: OK - load average: 11.83, 14.34, 14.82 [09:19:18] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:22:18] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:23:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:25:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.517578/1.75, alarm hl:np_load_avg=1.549805/2.0, alarm hl:mem_free=308.000000M/350M, alarm hl:available=1/0 [09:28:47] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:29:18] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:36:28] MySQL slave on thyme is WARNING: No slaves defined [09:38:57] Load avg. on willow is WARNING: WARNING - load average: 14.86, 15.44, 14.68 [09:43:23] س [09:45:18] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.198242/1.10, alarm hl:np_load_long=0.833008/1.55, alarm hl:mem_free=19055.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.198242/1.00, alarm hl:np_load_long=0.833008/1.50, alarm hl:mem_free=19055.000000M/350M, alarm hl:available=1/0 [09:46:18] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:51:18] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.130859/1.75, alarm hl:np_load_avg=2.211426/2.0, alarm hl:mem_free=536.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.130859/1.9, alarm hl:np_load_long=2.035645/2.25, alarm hl:mem_free=536.000000M/200M, alarm hl:available=1/0 [09:52:18] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:56:18] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.435059/1.75, alarm hl:np_load_avg=2.185059/2.0, alarm hl:mem_free=508.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.435059/1.9, alarm hl:np_load_long=2.047363/2.25, alarm hl:mem_free=508.000000M/200M, alarm hl:available=1/0 [10:03:18] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [10:03:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1255729.000000 [10:08:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36651.000000 [10:08:27] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:09:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:09:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:10:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:20:18] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:24:18] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:32:18] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.081055/1.00, alarm hl:np_load_long=0.832031/1.50, alarm hl:mem_free=19700.000000M/350M, alarm hl:available=1/0 [10:35:17] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [10:36:38] MySQL slave on thyme is WARNING: No slaves defined [10:38:57] Load avg. on willow is WARNING: WARNING - load average: 18.46, 16.64, 16.94 [10:43:18] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.902344/1.10, alarm hl:np_load_long=1.029297/1.55, alarm hl:mem_free=19038.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.902344/1.00, alarm hl:np_load_long=1.029297/1.50, alarm hl:mem_free=19038.000000M/350M, alarm hl:available=1/0 [10:51:55] Hello all [10:52:19] Good news: The enwp-import is done. During 1 hour it should be usable [10:53:18] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:57:58] Load avg. on willow is OK: OK - load average: 9.54, 12.40, 14.80 [11:00:19] Load avg. on adenia is WARNING: WARNING - load average: 18.88, 12.07, 7.33 [11:03:19] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [11:04:58] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1259389.000000 [11:08:18] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39928.000000 [11:09:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:09:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:10:27] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:11:18] Load avg. on adenia is OK: OK - load average: 8.82, 14.06, 11.95 [11:12:52] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [11:12:57] Load avg. on willow is WARNING: WARNING - load average: 15.88, 14.91, 14.61 [11:13:18] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.984863/1.75, alarm hl:np_load_avg=1.864258/2.0, alarm hl:mem_free=369.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.984863/1.9, alarm hl:np_load_long=1.826660/2.25, alarm hl:mem_free=369.000000M/200M, alarm hl:available=1/0 [11:13:57] Load avg. on willow is OK: OK - load average: 14.54, 14.72, 14.56 [11:17:19] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:18:37] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [11:20:17] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:21:37] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [11:24:18] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:24:18] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.053711/1.00, alarm hl:np_load_long=0.888672/1.50, alarm hl:mem_free=18815.000000M/350M, alarm hl:available=1/0 [11:26:18] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:27:17] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.811035/1.75, alarm hl:np_load_avg=1.727051/2.0, alarm hl:mem_free=368.000000M/350M, alarm hl:available=1/0 [11:29:52] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [11:31:54] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [11:33:54] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [12:03:07] Load avg. on willow is WARNING: WARNING - load average: 15.16, 14.90, 13.95 [12:03:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 469112.000000 [12:04:08] Load avg. on willow is OK: OK - load average: 13.39, 14.45, 13.85 [12:05:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1262993.000000 [12:05:31] @replag [12:05:32] Thehelpfulone: s1-rr-a: 2w 14h 50m 28s [+1.00 s/s]; s1-user: 2w 14h 50m 28s [+1.00 s/s]; s2-user: 56m 42s [-1.03 s/s]; s2-user-c: 11h 12m 3s [-1.60 s/s]; s5-user-c: 11h 12m 3s [-1.60 s/s] [12:09:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40048.000000 [12:09:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:09:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:10:47] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:18:39] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:20:39] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:21:39] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 465420 [12:24:38] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:51:39] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.889649/1.75, alarm hl:np_load_avg=1.717285/2.0, alarm hl:mem_free=422.000000M/350M, alarm hl:available=1/0 [12:52:38] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:55:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.079590/1.75, alarm hl:np_load_avg=1.790039/2.0, alarm hl:mem_free=329.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.079590/1.9, alarm hl:np_load_long=1.604004/2.25, alarm hl:mem_free=329.000000M/200M, alarm hl:available=1/0 [13:03:07] Load avg. on willow is WARNING: WARNING - load average: 16.52, 16.14, 14.15 [13:03:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 450562.000000 [13:06:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1266653.000000 [13:09:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36437.000000 [13:10:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:10:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:10:57] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:14:08] Load avg. on willow is OK: OK - load average: 11.47, 14.38, 14.57 [13:19:39] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [13:21:38] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:22:37] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 444341 [13:25:37] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:48:07] Load avg. on willow is WARNING: WARNING - load average: 15.24, 15.02, 14.66 [13:48:28] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.306152/1.00, alarm hl:np_load_long=0.382812/1.50, alarm hl:mem_free=329.000000M/350M, alarm hl:available=1/0 [13:49:06] @replag [13:49:07] DaBPunkt: s1-rr-a: error; s1-user: 2w 16h 34m 3s [-]; s2-user: 1h 15m 59s [-]; s2-user-c: 10h 29m 53s [-]; s5-user-c: 10h 29m 53s [-] [13:49:07] Load avg. on willow is OK: OK - load average: 12.24, 14.30, 14.43 [13:51:33] @replag [13:51:34] DaBPunkt: s1-rr-a: 5d 37m 34s [-]; s1-user: 2w 16h 36m 30s [+1.00 s/s]; s2-user: 1h 14m 59s [-0.41 s/s]; s2-user-c: 10h 21m 12s [-3.55 s/s]; s3-rr-a: 15s [-]; s3-user: 15s [-]; s5-user-c: 10h 21m 12s [-3.55 s/s] [13:52:40] @replag [13:52:40] DaBPunkt: s1-user: 2w 16h 37m 36s [-]; s2-user: 1h 13m 40s [-]; s2-user-c: 10h 13m 36s [-]; s5-user-c: 10h 13m 36s [-]; thyme: 5d 30m 12s [-] [13:54:56] SSH on adenia is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:00] @replag [13:55:00] DaBPunkt: s1-user: 2w 16h 39m 56s [+1.00 s/s]; s2-user: 1h 12m 49s [-0.36 s/s]; s2-user-c: 10h 9m 49s [-1.62 s/s]; s5-user-c: 10h 9m 49s [-1.62 s/s]; thyme: 5d 24m 0s [-2.65 s/s] [13:55:17] /tmp on wolfsbane is WARNING: DISK WARNING - free space: /tmp 676 MB (16% inode=99%): [13:55:48] SSH on adenia is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [13:56:16] /tmp on wolfsbane is OK: DISK OK - free space: /tmp 1076 MB (23% inode=99%): [13:58:55] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [13:59:07] Load avg. on willow is WARNING: WARNING - load average: 15.17, 15.04, 14.76 [13:59:28] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [14:03:48] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:04:19] @replag [14:04:20] DaBPunkt: s1-rr-a: 4d 23h 56m 5s [-]; s1-user: 2w 16h 49m 16s [-]; s2-user: 1h 12m 26s [-]; s2-user-c: 9h 37m 52s [-]; s5-user-c: 9h 37m 52s [-] [14:04:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 431638.000000 [14:06:12] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1270261.000000 [14:08:22] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:08:25] @replag [14:08:27] IWorld: s1-rr-a: 4d 23h 39m 6s [-4.16 s/s]; s1-user: 2w 16h 53m 21s [+1.00 s/s]; s2-user: 1h 11m 57s [-0.12 s/s]; s2-user-c: 9h 25m 48s [-2.95 s/s]; s5-user-c: 9h 25m 49s [-2.93 s/s] [14:09:23] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33783.000000 [14:10:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:10:55] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:11:03] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:15:56] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=3.243164/1.75, alarm hl:np_load_avg=2.464844/2.0, alarm hl:mem_free=262.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=3.243164/1.9, alarm hl:np_load_long=2.210938/2.25, alarm hl:mem_free=262.000000M/200M, alarm hl:available=1/0 [14:19:57] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [14:21:55] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:22:56] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6989&oldid=6960&rcid=9201 * 122.248.163.3 * (+13) () [14:22:56] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 411980 [14:23:25] [[Wiki server assignments]] M 10https://wiki.toolserver.org/w/index.php?diff=6990&oldid=6989&rcid=9202 * Dab * (-13) (Reverted edits by [[Special:Contributions/122.248.163.3|122.248.163.3]] ([[User talk:122.248.163.3|talk]]) to last revision by [[User:91.198.174.202|91.198.174.202]]) [14:24:52] 3(commented) [ACCAPP-485] UTRS developer <10https://jira.toolserver.org/browse/ACCAPP-485> (Chris Howie) [14:24:57] 3(commented) [ACCAPP-460] I want run a robot for fawiki, and im want mostly to faster doing job. <10https://jira.toolserver.org/browse/ACCAPP-460> (Mahdi haji) [14:25:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:27:02] DaB. * [Toolserver-l] News about s1/enwiki [14:34:41] @replag [14:34:42] IWorld: s1-rr-a: 4d 21h 38m 39s [-4.58 s/s]; s1-user: 2w 17h 19m 38s [+1.00 s/s]; s2-user: 1h 7m 27s [-0.17 s/s]; s2-user-c: 8h 14m 22s [-2.72 s/s]; s5-user-c: 8h 14m 22s [-2.72 s/s] [14:38:54] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:41:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.105469/1.75, alarm hl:np_load_avg=2.100586/2.0, alarm hl:mem_free=498.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.105469/1.9, alarm hl:np_load_long=2.121582/2.25, alarm hl:mem_free=498.000000M/200M, alarm hl:available=1/0 [14:49:57] @replag [14:49:58] DaBPunkt: s1-rr-a: 4d 20h 26m 57s [-4.70 s/s]; s1-user: 2w 17h 34m 53s [+1.00 s/s]; s2-user: 1h 5m 0s [-0.16 s/s]; s2-user-c: 7h 23m 46s [-3.32 s/s]; s5-user-c: 7h 23m 46s [-3.32 s/s]; s6-rr-a: 51s [-]; s6-user: 51s [-] [14:52:55] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:59:12] Load avg. on willow is WARNING: WARNING - load average: 15.52, 15.87, 16.48 [15:05:56] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 415055.000000 [15:06:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1273861.000000 [15:09:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23243.000000 [15:09:54] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.345703/1.10, alarm hl:np_load_long=0.737305/1.55, alarm hl:mem_free=20422.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.345703/1.00, alarm hl:np_load_long=0.737305/1.50, alarm hl:mem_free=20422.000000M/350M, alarm hl:available=1/0 [15:11:13] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:11:54] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:11:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:11:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:20:54] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [15:22:03] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:23:55] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 411103 [15:26:55] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:33:02] Dispenser: remember that error on qcronsub with the LC_ALL case? I'm still with the same error :s [15:39:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.000977/1.75, alarm hl:np_load_avg=2.022461/2.0, alarm hl:mem_free=1085.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.000977/1.9, alarm hl:np_load_long=2.012695/2.25, alarm hl:mem_free=1085.000000M/200M, alarm hl:available=1/0 [15:59:13] Load avg. on willow is WARNING: WARNING - load average: 15.62, 15.89, 16.44 [16:03:52] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.233399/1.10, alarm hl:np_load_long=0.803711/1.55, alarm hl:mem_free=21008.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.233399/1.00, alarm hl:np_load_long=0.803711/1.50, alarm hl:mem_free=21008.000000M/350M, alarm hl:available=1/0 [16:04:54] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [16:06:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1277461.000000 [16:06:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 397933.000000 [16:07:02] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 113381 MB (11% inode=99%): [16:08:03] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:09:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20809.000000 [16:12:02] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:12:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:12:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:13:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.471191/1.75, alarm hl:np_load_avg=2.259277/2.0, alarm hl:mem_free=303.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.471191/1.9, alarm hl:np_load_long=2.163086/2.25, alarm hl:mem_free=303.000000M/200M, alarm hl:available=1/0 [16:21:54] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [16:23:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:24:53] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 391013 [16:27:03] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:30:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:33:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.846680/1.75, alarm hl:np_load_avg=2.052734/2.0, alarm hl:mem_free=512.000000M/350M, alarm hl:available=1/0 [16:35:02] [[Comparison of osm2pgsql and osmosis for GeoShape]] ! 10https://wiki.toolserver.org/w/index.php?diff=6991&oldid=6986&rcid=9203 * 84.112.160.35 * (-242) (/* Osm2pgsql features */ -setup) [16:52:13] Load avg. on willow is OK: OK - load average: 10.83, 13.82, 14.84 [16:57:13] Load avg. on willow is WARNING: WARNING - load average: 16.24, 15.00, 15.01 [16:58:53] 3(commented) [UTRS-94] Fields are truncated <10https://jira.toolserver.org/browse/UTRS-94> (Martijn Hoekstra) [16:58:55] 3(created) [UTRS-94] Fields are truncated; UTRS: Main Interface; Minor Bug <10https://jira.toolserver.org/browse/UTRS-94> (Martijn Hoekstra) [17:06:15] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1281065.000000 [17:07:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 369980.000000 [17:09:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17571.000000 [17:12:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:12:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:12:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:13:15] @replag [17:13:15] Dispenser: s1-rr-a: 4d 6h 7m 13s [-6.00 s/s]; s1-user: 2w 19h 58m 11s [+1.00 s/s]; s2-user: 31m 55s [-0.23 s/s]; s2-user-c: 4h 52m 57s [-1.05 s/s]; s3-rr-a: 1m 59s [-]; s3-user: 1m 59s [-]; s5-user-c: 4h 52m 50s [-1.05 s/s] [17:14:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.067383/1.75, alarm hl:np_load_avg=2.215820/2.0, alarm hl:mem_free=336.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.067383/1.9, alarm hl:np_load_long=2.119141/2.25, alarm hl:mem_free=336.000000M/200M, alarm hl:available=1/0 [17:14:31] Alchimista: I don't remember anything about qcronsub [17:20:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:22:52] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [17:23:02] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.000976/1.00, alarm hl:np_load_long=0.768555/1.50, alarm hl:mem_free=20904.000000M/350M, alarm hl:available=1/0 [17:23:13] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:24:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:24:52] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 362699 [17:27:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:27:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.931152/1.75, alarm hl:np_load_avg=1.939941/2.0, alarm hl:mem_free=510.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.931152/1.9, alarm hl:np_load_long=2.006836/2.25, alarm hl:mem_free=510.000000M/200M, alarm hl:available=1/0 [17:29:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:51:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.310547/1.75, alarm hl:np_load_avg=2.048340/2.0, alarm hl:mem_free=218.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.310547/1.9, alarm hl:np_load_long=1.987793/2.25, alarm hl:mem_free=218.000000M/200M, alarm hl:available=1/0 [17:57:24] Load avg. on willow is WARNING: WARNING - load average: 12.17, 14.65, 15.28 [18:04:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:06:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1284667.000000 [18:07:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 349067.000000 [18:09:33] Load avg. on willow is OK: OK - load average: 10.82, 13.81, 14.95 [18:09:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16863.000000 [18:12:33] Load avg. on willow is WARNING: WARNING - load average: 15.57, 14.99, 15.23 [18:12:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:13:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:13:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:13:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.914551/1.75, alarm hl:np_load_avg=1.872070/2.0, alarm hl:mem_free=508.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.914551/1.9, alarm hl:np_load_long=1.902832/2.25, alarm hl:mem_free=508.000000M/200M, alarm hl:available=1/0 [18:15:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:17:33] Load avg. on willow is OK: OK - load average: 13.12, 14.25, 14.87 [18:19:02] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.840820/1.10, alarm hl:np_load_long=1.015625/1.55, alarm hl:mem_free=20329.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.840820/1.00, alarm hl:np_load_long=1.015625/1.50, alarm hl:mem_free=20329.000000M/350M, alarm hl:available=1/0 [18:22:54] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [18:23:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:23:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:24:53] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 343830 [18:26:03] @replag [18:26:03] euphoria: s1-rr-a: 3d 23h 25m 30s [-5.52 s/s]; s1-user: 2w 21h 10m 59s [+1.00 s/s]; s2-user: 1h 3m 35s [+0.44 s/s]; s2-user-c: 4h 38m 11s [-0.20 s/s]; s5-user-c: 4h 38m 11s [-0.20 s/s] [18:27:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:32:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.019043/1.75, alarm hl:np_load_avg=1.962402/2.0, alarm hl:mem_free=348.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.019043/1.9, alarm hl:np_load_long=1.873047/2.25, alarm hl:mem_free=348.000000M/200M, alarm hl:available=1/0 [18:32:33] Load avg. on willow is WARNING: WARNING - load average: 15.30, 15.55, 14.96 [18:38:34] Load avg. on willow is OK: OK - load average: 11.23, 14.05, 14.57 [18:45:02] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.243164/1.10, alarm hl:np_load_long=0.964844/1.55, alarm hl:mem_free=19656.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.243164/1.00, alarm hl:np_load_long=0.964844/1.50, alarm hl:mem_free=19656.000000M/350M, alarm hl:available=1/0 [18:46:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:50:33] Load avg. on willow is WARNING: WARNING - load average: 24.43, 17.35, 15.59 [19:06:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1288270.000000 [19:07:36] @reag [19:07:47] @replag [19:07:48] matthewrbowker: s1-rr-a: 3d 20h 32m 50s [-4.14 s/s]; s1-user: 2w 21h 52m 43s [+1.00 s/s]; s2-user: 1h 22m 17s [+0.45 s/s]; s2-user-c: 4h 32m 2s [-0.15 s/s]; s5-user-c: 4h 32m 2s [-0.15 s/s] [19:07:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 333177.000000 [19:09:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16361.000000 [19:11:46] ooh it's getting better :) [19:11:56] well done DaBPunkt :) [19:12:33] although s1-user isn't so happy [19:12:41] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:13:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:13:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:14:55] Thehelpfulone: s1-user will be switched to thyme (currently s1-rr-a) when it's caught up [19:15:09] ok [19:17:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:23:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.861328/1.75, alarm hl:np_load_avg=1.854492/2.0, alarm hl:mem_free=448.000000M/350M, alarm hl:available=1/0 [19:23:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:23:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [19:24:53] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 327318 [19:27:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:28:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:32:34] Load avg. on willow is OK: OK - load average: 10.39, 13.63, 14.75 [19:42:34] Load avg. on willow is WARNING: WARNING - load average: 16.00, 15.14, 14.86 [19:43:34] Load avg. on willow is OK: OK - load average: 14.62, 14.84, 14.76 [19:51:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.068848/1.75, alarm hl:np_load_avg=1.936035/2.0, alarm hl:mem_free=230.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.068848/1.9, alarm hl:np_load_long=1.889160/2.25, alarm hl:mem_free=230.000000M/200M, alarm hl:available=1/0 [19:55:14] [[Special:Log/newusers]] create 10 * Célestelove37 * (New user account) [19:56:53] 3(assigned) [UTRS-83] Comments cut off in appeals <10https://jira.toolserver.org/browse/UTRS-83> (TParis) [19:56:55] 3(assigned) [UTRS-94] Fields are truncated <10https://jira.toolserver.org/browse/UTRS-94> (TParis) [19:56:59] 3(resolved) [UTRS-83] Comments cut off in appeals <10https://jira.toolserver.org/browse/UTRS-83> (TParis) [19:58:54] 3(commented) [UTRS-94] Fields are truncated <10https://jira.toolserver.org/browse/UTRS-94> (TParis) [20:00:52] 3(created) [UTRS-95] Blocking admin on appeal form; UTRS; Improvement <10https://jira.toolserver.org/browse/UTRS-95> (TParis) [20:01:02] DaB. * Re: [Toolserver-announce] [Toolserver-l] Corruption of s6 and (short) downtime of s6 tomorrow (Tuesday) [20:04:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:06:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1291873.000000 [20:07:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 306536.000000 [20:08:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.852539/1.75, alarm hl:np_load_avg=1.937988/2.0, alarm hl:mem_free=771.000000M/350M, alarm hl:available=1/0 [20:10:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:10:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12702.000000 [20:12:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:13:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:13:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:13:34] Load avg. on willow is WARNING: WARNING - load average: 15.64, 15.69, 15.34 [20:17:33] Load avg. on willow is OK: OK - load average: 12.44, 14.46, 14.96 [20:18:52] 3(assigned) [SWMTBOT-45] Make SWMTBot recongnize commands based on its current nickname instead of its set nickname in SWMTBot.ini <10https://jira.toolserver.org/browse/SWMTBOT-45> (Daniel Salciccioli) [20:18:52] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:20:54] 3(work stopped) [SWMTBOT-45] Make SWMTBot recongnize commands based on its current nickname instead of its set nickname in SWMTBot.ini <10https://jira.toolserver.org/browse/SWMTBOT-45> (Daniel Salciccioli) [20:20:56] 3(work started) [SWMTBOT-45] Make SWMTBot recongnize commands based on its current nickname instead of its set nickname in SWMTBot.ini <10https://jira.toolserver.org/browse/SWMTBOT-45> (Daniel Salciccioli) [20:23:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:24:03] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.095703/1.00, alarm hl:np_load_long=0.889648/1.50, alarm hl:mem_free=19779.000000M/350M, alarm hl:available=1/0 [20:24:52] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:25:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:25:52] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 297450 [20:26:32] Load avg. on willow is WARNING: WARNING - load average: 18.32, 15.68, 15.10 [20:27:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:30:52] 3(created) [SWMTBOT-46] Support for MySQL; SWMTBot; New Feature <10https://jira.toolserver.org/browse/SWMTBOT-46> (Krinkle) [20:32:56] 3(updated) [SWMTBOT-20] Implement a way to share a (MySQL) database with multiple bots <10https://jira.toolserver.org/browse/SWMTBOT-20> (Krinkle) [20:34:55] 3(assigned) [SWMTBOT-46] Support for MySQL <10https://jira.toolserver.org/browse/SWMTBOT-46> (Daniel Salciccioli) [20:36:53] 3(commented) [SWMTBOT-46] Support for MySQL <10https://jira.toolserver.org/browse/SWMTBOT-46> (Daniel Salciccioli) [20:42:35] Load avg. on willow is OK: OK - load average: 12.88, 14.23, 14.93 [20:44:02] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.083008/1.00, alarm hl:np_load_long=0.883789/1.50, alarm hl:mem_free=19854.000000M/350M, alarm hl:available=1/0 [20:46:36] Load avg. on willow is WARNING: WARNING - load average: 16.79, 15.66, 15.35 [20:49:11] Does anyone know a way I can make it so that I don't have to to `source ~/.profile` every time I do `become cvn` ? [20:49:37] when I'm in the cvn mmt my own .profile no longer applies but neither does cvn's [20:56:09] add ". ~/.profile" to .bashrc [20:56:18] (assuming you use bash) [20:56:25] (at least, that's what works for me) [20:57:35] Load avg. on willow is OK: OK - load average: 11.26, 14.05, 14.94 [20:59:05] @replag [20:59:05] Thehelpfulone: s1-rr-a: 3d 6h 53m 31s [-7.36 s/s]; s1-user: 2w 23h 44m 1s [+1.00 s/s]; s2-user: 1h 41m 37s [+0.17 s/s]; s2-user-c: 3h 34m 10s [-0.52 s/s]; s5-user-c: 3h 34m 10s [-0.52 s/s] [21:00:52] 3(created) [SWMTBOT-47] Implement support for master/slave relationships between bots that use the same (MySQL) database; SWMTBot; New Feature <10https://jira.toolserver.org/browse/SWMTBOT-47> (Krinkle) [21:03:52] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [21:06:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1295484.000000 [21:07:58] 3(created) [CVN-6] Merge toolserver "SWMTBOT" project into "CVN"; CVN; Task <10https://jira.toolserver.org/browse/CVN-6> (Krinkle) [21:07:59] 3(commented) [CVN-6] Merge toolserver "SWMTBOT" project into "CVN" <10https://jira.toolserver.org/browse/CVN-6> (Krinkle) [21:08:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 280473.000000 [21:10:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12531.000000 [21:12:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:13:12] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.225586/1.10, alarm hl:np_load_long=0.758789/1.55, alarm hl:mem_free=19442.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.225586/1.00, alarm hl:np_load_long=0.758789/1.50, alarm hl:mem_free=19442.000000M/350M, alarm hl:available=1/0 [21:13:34] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:13:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:14:12] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:19:43] Load avg. on willow is WARNING: WARNING - load average: 14.03, 14.80, 15.36 [21:23:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:24:43] Load avg. on willow is OK: OK - load average: 13.66, 14.22, 14.98 [21:24:52] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:25:52] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 275893 [21:27:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:27:43] Load avg. on willow is WARNING: WARNING - load average: 15.39, 15.10, 15.22 [21:36:44] Load avg. on willow is OK: OK - load average: 12.19, 14.40, 14.97 [21:44:52] 3(commented) [ACCAPP-461] Analysing the development of Bots <10https://jira.toolserver.org/browse/ACCAPP-461> (Franz Herbach) [21:49:52] 3(updated) [ACCAPP-461] Analysing the development of Bots <10https://jira.toolserver.org/browse/ACCAPP-461> (Franz Herbach) [21:59:19] @replag [21:59:20] Joan: s1-rr-a: 3d 1h 46m 58s [-5.09 s/s]; s1-user: 2w 1d 44m 15s [+1.00 s/s]; s2-user: 1h 55m 32s [+0.23 s/s]; s2-user-c: 3h 1m 43s [-0.54 s/s]; s5-user-c: 3h 1m 43s [-0.54 s/s] [21:59:24] \o/ [22:00:54] 3(created) [ACCAPP-487] Analysing the development of Bots; Account Approval; Blocker New Account <10https://jira.toolserver.org/browse/ACCAPP-487> (Franz Herbach) [22:02:42] Load avg. on willow is WARNING: WARNING - load average: 15.16, 15.89, 15.19 [22:07:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1299143.000000 [22:09:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 265220.000000 [22:11:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11410.000000 [22:13:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:14:30] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:14:30] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:17:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.884766/1.75, alarm hl:np_load_avg=2.009277/2.0, alarm hl:mem_free=802.000000M/350M, alarm hl:available=1/0 [22:19:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:21:52] 3(created) [DBQ-179] Database Query; Database Queries; Critical Task <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [22:22:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.816406/1.75, alarm hl:np_load_avg=1.917480/2.0, alarm hl:mem_free=855.000000M/350M, alarm hl:available=1/0 [22:24:41] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:25:22] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [22:26:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 258906 [22:28:31] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:34:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:37:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 106859 MB (10% inode=99%): [22:38:30] replag on thyme is now below 3 days -- yay! [22:40:19] :O [22:40:53] 3(commented) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Hoo man) [22:45:20] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.329101/1.10, alarm hl:np_load_long=0.701172/1.55, alarm hl:mem_free=20374.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.329101/1.00, alarm hl:np_load_long=0.701172/1.50, alarm hl:mem_free=20374.000000M/350M, alarm hl:available=1/0 [22:47:22] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [22:53:51] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:54:02] Alchimista * [Toolserver-l] UnicodeEncodeError [22:54:17] Dispenser: i thought i've talked to you :P i've sended a mail to the list. [22:54:52] Load avg. on willow is OK: OK - load average: 13.16, 13.89, 14.95 [22:57:27] @replag [22:57:28] Joan: s1-rr-a: 2d 22h 31m 11s [-3.37 s/s]; s1-user: 2w 1d 1h 42m 24s [+1.00 s/s]; s2-user: 1h 43m 46s [-0.20 s/s]; s2-user-c: 2h 57m 38s [-0.07 s/s]; s5-user-c: 2h 57m 38s [-0.07 s/s] [22:57:53] Load avg. on willow is WARNING: WARNING - load average: 13.99, 14.57, 15.09 [22:59:52] Load avg. on willow is OK: OK - load average: 11.04, 13.59, 14.68 [23:04:57] 3(updated) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [23:07:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1302744.000000 [23:09:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 249861.000000 [23:11:55] 3(commented) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Hoo man) [23:12:01] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9587.000000 [23:13:21] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:13:40] Alchimista: UnicodeEncodeError with wikipedia.output() occurs with byte strings, use Unicode strings instead [23:13:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.846680/1.75, alarm hl:np_load_avg=1.943848/2.0, alarm hl:mem_free=169.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.846680/1.9, alarm hl:np_load_long=1.894531/2.25, alarm hl:mem_free=169.000000M/200M, alarm hl:available=1/0 [23:13:47] source code? [23:14:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:14:40] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:14:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:15:16] Dispenser: with output() i'm not sure if an error occours, seems to miss some outputs from there. The code is adapted from: http://code.google.com/p/avbot/source/browse/#svn%2Ftrunk [23:16:54] it runs perfectly on willow, if i runnit just like a regular non-long run job [23:43:59] 3(commented) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [23:45:52] 3(work started) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Hoo man)