[00:00:01] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3549.000000 [00:00:18] @replag [00:00:18] DaBPunkt: s1-rr-a: 2d 15h 18m 50s [-6.88 s/s]; s1-user: 2w 1d 2h 45m 14s [+1.00 s/s]; s2-user: 57m 44s [-0.73 s/s]; s2-user-c: 56m 53s [-1.92 s/s]; s3-rr-a: 10s [-0.00 s/s]; s3-user: 10s [-0.00 s/s]; s5-user-c: 56m 53s [-1.92 s/s] [00:00:53] I will now switch over sql-s1-user [00:03:54] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [00:04:32] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 108503 MB (11% inode=99%): [00:07:03] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1519.000000 [00:07:41] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [00:07:55] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [00:10:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [00:10:20] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [00:10:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 224037.000000 [00:14:10] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:14:40] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:14:41] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:15:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.759766/1.75, alarm hl:np_load_avg=2.300781/2.0, alarm hl:mem_free=343.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.759766/1.9, alarm hl:np_load_long=2.179199/2.25, alarm hl:mem_free=343.000000M/200M, alarm hl:available=1/0 [00:16:02] Load avg. on willow is WARNING: WARNING - load average: 18.76, 17.96, 17.31 [00:21:50] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41041 MB (10% inode=99%): [00:24:50] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 68687 MB (16% inode=99%): [00:25:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:26:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 218807 [00:26:50] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [00:28:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:29:40] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:32:40] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.031738/1.75, alarm hl:np_load_avg=2.124512/2.0, alarm hl:mem_free=472.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.031738/1.9, alarm hl:np_load_long=2.088379/2.25, alarm hl:mem_free=472.000000M/200M, alarm hl:available=1/0 [00:34:50] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39692 MB (9% inode=99%): [00:41:50] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 80050 MB (19% inode=99%): [00:42:00] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2140.000000 [00:42:20] MySQL on rosemary is OK: Uptime: 1679 Threads: 1 Questions: 8 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 11 Queries per second avg: 0.4 [00:49:59] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [00:52:20] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [00:53:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:56:21] MySQL on rosemary is OK: Uptime: 246 Threads: 2 Questions: 10 Slow queries: 0 Opens: 15 Flush tables: 1 Open tables: 8 Queries per second avg: 0.40 [00:57:00] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3040.000000 [00:58:30] /sql on rosemary is OK: DISK OK - free space: /sql 655265 MB (70% inode=99%): [01:00:49] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 40335 MB (9% inode=99%): [01:00:56] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [01:07:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [01:07:52] 3(commented) [MNT-1227] Re-Import of enwiki <10https://jira.toolserver.org/browse/MNT-1227> (DaB.) [01:09:50] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 60680 MB (14% inode=99%): [01:10:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208609.000000 [01:10:44] OK Tried to do an archive.org to Commons transfer with URL2Commons [01:10:52] And it gave no error [01:10:58] The file isn't it at Commons [01:11:01] Suggestions? [01:13:24] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:14:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:15:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:15:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:16:09] Load avg. on willow is WARNING: WARNING - load average: 15.05, 17.32, 17.67 [01:17:51] nacht ts [01:18:00] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:18:02] John * Re: [Toolserver-l] UnicodeEncodeError [01:21:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4484.000000 [01:23:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.796387/1.75, alarm hl:np_load_avg=1.838379/2.0, alarm hl:mem_free=471.000000M/350M, alarm hl:available=1/0 [01:25:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:26:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 205596 [01:27:10] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [01:28:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:33:59] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:47:11] Load avg. on willow is OK: OK - load average: 10.54, 13.72, 14.87 [02:07:11] Load avg. on willow is WARNING: WARNING - load average: 17.12, 15.73, 14.89 [02:08:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [02:10:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 197804.000000 [02:14:31] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:15:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:15:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:21:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8085.000000 [02:24:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.452149/1.75, alarm hl:np_load_avg=1.889649/2.0, alarm hl:mem_free=158.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.452149/1.9, alarm hl:np_load_long=1.988770/2.25, alarm hl:mem_free=158.000000M/200M, alarm hl:available=1/0 [02:26:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:27:10] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [02:27:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 193413 [02:28:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:31:47] @replag [02:31:48] Joan: s1-rr-a: 2d 5h 30m 56s [-3.88 s/s]; s1-user: 2d 5h 30m 56s [-122.47 s/s]; s2-user-c: 21s [-0.37 s/s]; s5-user-c: 21s [-0.37 s/s]; s6-rr-a: 14s [-0.00 s/s]; s6-user: 14s [-0.00 s/s] [02:43:42] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.187500/1.10, alarm hl:np_load_long=0.794922/1.55, alarm hl:mem_free=19260.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.187500/1.00, alarm hl:np_load_long=0.794922/1.50, alarm hl:mem_free=19260.000000M/350M, alarm hl:available=1/0 [02:45:41] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:07:11] Load avg. on willow is WARNING: WARNING - load average: 15.84, 16.75, 16.45 [03:08:51] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [03:10:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 183475.000000 [03:15:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:15:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:15:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:21:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11685.000000 [03:24:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:25:10] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1871.000000 [03:26:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:27:10] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1796.000000 [03:27:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 180244 [03:28:10] Load avg. on willow is OK: OK - load average: 10.41, 13.14, 14.80 [03:28:10] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:28:50] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:43:52] 3(resolved) [ACC-206] MySQL error on sandbox <10https://jira.toolserver.org/browse/ACC-206> (Simon Walker) [03:46:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.903320/1.75, alarm hl:np_load_avg=1.788086/2.0, alarm hl:mem_free=327.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.903320/1.9, alarm hl:np_load_long=1.777832/2.25, alarm hl:mem_free=327.000000M/200M, alarm hl:available=1/0 [03:47:00] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:53:53] 3(commented) [ACC-240] Can the list handle spoofs? <10https://jira.toolserver.org/browse/ACC-240> (Simon Walker) [03:55:55] 3(resolved) [ACC-240] Can the list handle spoofs? <10https://jira.toolserver.org/browse/ACC-240> (Simon Walker) [03:58:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.774414/1.75, alarm hl:np_load_avg=1.769531/2.0, alarm hl:mem_free=567.000000M/350M, alarm hl:available=1/0 [04:02:11] Load avg. on willow is WARNING: WARNING - load average: 13.53, 15.95, 14.90 [04:02:37] Hello. [04:02:38] I'm here. [04:02:41] @replag [04:02:42] Joan: s1-rr-a: 2d 25m 33s [-3.36 s/s]; s1-user: 2d 25m 33s [-3.36 s/s]; s3-rr-a: 25s [+0.00 s/s]; s3-user: 25s [+0.00 s/s] [04:02:52] 3(updated) [ACC-192] Triage should check editcount of existing users <10https://jira.toolserver.org/browse/ACC-192> (Simon Walker) [04:02:55] 3(commented) [ACC-108] Reason for suspension. <10https://jira.toolserver.org/browse/ACC-108> (Simon Walker) [04:03:10] Load avg. on willow is OK: OK - load average: 10.43, 14.68, 14.52 [04:08:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [04:10:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 172838.000000 [04:15:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:15:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:16:00] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:22:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15346.000000 [04:26:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:27:20] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 170181 [04:28:21] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:28:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:30:20] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6992&oldid=6990&rcid=9205 * 91.198.174.202 * (+1) (updated page) [04:33:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.329101/1.75, alarm hl:np_load_avg=1.671875/2.0, alarm hl:mem_free=185.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.329101/1.9, alarm hl:np_load_long=1.612793/2.25, alarm hl:mem_free=185.000000M/200M, alarm hl:available=1/0 [04:52:10] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:55:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.860351/1.75, alarm hl:np_load_avg=1.711914/2.0, alarm hl:mem_free=289.000000M/350M, alarm hl:available=1/0 [05:08:52] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [05:10:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 164495.000000 [05:12:11] Load avg. on willow is WARNING: WARNING - load average: 15.97, 15.25, 14.21 [05:13:10] Load avg. on willow is OK: OK - load average: 14.03, 14.88, 14.15 [05:15:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:15:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:16:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:17:10] Load avg. on willow is WARNING: WARNING - load average: 17.23, 17.24, 15.33 [05:22:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18948.000000 [05:27:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:27:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 162399 [05:28:21] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:28:51] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:29:00] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:29:10] Load avg. on willow is OK: OK - load average: 9.70, 13.47, 14.61 [05:33:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.965332/1.75, alarm hl:np_load_avg=2.545898/2.0, alarm hl:mem_free=250.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.965332/1.9, alarm hl:np_load_long=2.161133/2.25, alarm hl:mem_free=250.000000M/200M, alarm hl:available=1/0 [05:51:10] Load avg. on willow is WARNING: WARNING - load average: 19.99, 18.70, 17.76 [06:00:10] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [06:01:09] Load avg. on willow is WARNING: WARNING - load average: 24.64, 19.34, 17.82 [06:09:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [06:11:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 157224.000000 [06:14:21] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1929.000000 [06:15:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:15:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:15:43] @replag [06:15:44] Dispenser: s1-rr-a: 1d 19h 31m 29s [-2.21 s/s]; s1-user: 1d 19h 31m 29s [-2.21 s/s]; s2-user: 2m 24s [-0.15 s/s]; s2-user-c: 33m 27s [+0.15 s/s]; s3-rr-a: 26s [+0.00 s/s]; s3-user: 26s [+0.00 s/s]; s5-user-c: 33m 27s [+0.15 s/s] [06:16:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:22:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22548.000000 [06:28:21] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:28:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:28:21] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 155393 [06:29:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:33:00] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.931641/1.75, alarm hl:np_load_avg=2.143066/2.0, alarm hl:mem_free=351.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.931641/1.9, alarm hl:np_load_long=2.242188/2.25, alarm hl:mem_free=351.000000M/200M, alarm hl:available=1/0 [06:49:02] DeltaQuad * Re: [Toolserver-l] UnicodeEncodeError [06:57:20] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1704.000000 [06:58:01] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.205078/1.10, alarm hl:np_load_long=1.343750/1.55, alarm hl:mem_free=20240.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.205078/1.00, alarm hl:np_load_long=1.343750/1.50, alarm hl:mem_free=20240.000000M/350M, alarm hl:available=1/0 [07:00:01] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:00:10] SMF on willow is OK: OK - all services online [07:01:20] Load avg. on willow is WARNING: WARNING - load average: 23.72, 20.71, 19.22 [07:03:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:09:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [07:11:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 146715.000000 [07:15:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:15:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:22:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26151.000000 [07:28:32] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 143087 [07:28:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:29:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:29:20] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:33:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.163086/1.75, alarm hl:np_load_avg=2.366699/2.0, alarm hl:mem_free=381.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.163086/1.9, alarm hl:np_load_long=2.313965/2.25, alarm hl:mem_free=381.000000M/200M, alarm hl:available=1/0 [07:48:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:58:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [07:58:41] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [08:01:02] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.667480/1.75, alarm hl:np_load_avg=2.239746/2.0, alarm hl:mem_free=420.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.667480/1.9, alarm hl:np_load_long=2.218750/2.25, alarm hl:mem_free=420.000000M/200M, alarm hl:available=1/0 [08:01:19] Load avg. on willow is WARNING: WARNING - load average: 20.82, 17.98, 17.77 [08:03:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:09:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [08:11:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134450.000000 [08:15:10] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:15:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:17:02] Merlijn van Deen * Re: [Toolserver-l] UnicodeEncodeError [08:22:10] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29751.000000 [08:23:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:29:20] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:29:31] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 130706 [08:29:31] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:30:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:37:42] @replag [08:37:43] IWorld: s1-rr-a: 1d 12h 3m 22s [-3.16 s/s]; s1-user: 1d 12h 3m 22s [-3.16 s/s]; s2-user: 42m 58s [+0.29 s/s]; s2-user-c: 14m 56s [-0.13 s/s]; s3-rr-a: 28s [+0.00 s/s]; s3-user: 28s [+0.00 s/s]; s5-user-c: 14m 56s [-0.13 s/s] [08:38:04] @list [08:38:04] IWorld: Admin, Channel, Config, JobStats, Misc, Network, Owner, Scheduler, Services, TSReplag, Time, User, and Utilities [08:38:10] :-) [08:38:18] @list TSReplag [08:38:18] IWorld: replag [08:39:22] @uptime [08:39:26] meeh [08:46:01] [[Special:Log/newusers]] create 10 * Frank.messelis * (New user account) [08:47:10] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:47:19] Load avg. on willow is OK: OK - load average: 11.15, 13.79, 14.83 [08:57:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.684082/1.75, alarm hl:np_load_avg=1.602539/2.0, alarm hl:mem_free=217.000000M/350M, alarm hl:available=1/0 [09:02:21] Load avg. on willow is WARNING: WARNING - load average: 16.03, 16.39, 15.07 [09:03:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:10:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [09:11:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126037.000000 [09:15:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:15:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:22:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33350.000000 [09:29:20] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:29:30] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 122795 [09:29:30] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:30:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:37:20] Load avg. on willow is OK: OK - load average: 12.89, 13.94, 14.95 [09:42:20] Load avg. on willow is WARNING: WARNING - load average: 16.75, 15.27, 15.18 [09:44:12] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.021484/1.00, alarm hl:np_load_long=0.622070/1.50, alarm hl:mem_free=19060.000000M/350M, alarm hl:available=1/0 [09:45:11] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:54:20] Load avg. on willow is OK: OK - load average: 13.71, 14.64, 14.94 [09:57:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.768066/1.75, alarm hl:np_load_avg=1.817871/2.0, alarm hl:mem_free=259.000000M/350M, alarm hl:available=1/0 [10:01:20] Load avg. on willow is WARNING: WARNING - load average: 20.27, 17.05, 15.72 [10:03:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:04:10] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:07:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.037598/1.75, alarm hl:np_load_avg=1.977051/2.0, alarm hl:mem_free=232.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.037598/1.9, alarm hl:np_load_long=1.933105/2.25, alarm hl:mem_free=232.000000M/200M, alarm hl:available=1/0 [10:10:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [10:11:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117124.000000 [10:15:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:16:00] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:16:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:18:19] Load avg. on willow is OK: OK - load average: 9.36, 13.02, 14.67 [10:22:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36950.000000 [10:22:54] 3(commented) [ACCAPP-468] Rcsprinter <10https://jira.toolserver.org/browse/ACCAPP-468> (David Moon) [10:23:50] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:29:19] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:30:06] [[Special:Log/newusers]] create 10 * Vis met 1 oog * (New user account) [10:30:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:30:30] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113349 [10:30:30] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:32:19] Load avg. on willow is WARNING: WARNING - load average: 15.50, 13.12, 13.16 [10:33:31] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:34:20] Load avg. on willow is OK: OK - load average: 14.05, 13.57, 13.34 [10:34:44] http://toolserver.org/~magnus/url2commons.php is broken [10:35:03] It doesn't give an error, but failed to upload the file I gave it [10:40:39] Either fix the tool or delete it from toolserver [10:50:43] hi, I need some help with running a tool on toolserver [10:52:51] I've been given an account on the toolserver and logged in using PuTTY... now what? [11:02:12] ah, DaBPunkt. can you help me? [11:02:27] Maybe? ;) [11:02:42] I've been given an account on the toolserver and logged in using PuTTY... now what? [11:04:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:04:24] In your account-requesr you wrote that you paln to run a bot that reverts test-edits – I guess you should do that, shouldn't you? [11:04:40] don't know how though from the putty [11:04:59] do I link it to the code somehow? [11:06:24] I am very new to the toolserver [11:08:23] did you read https://wiki.toolserver.org/view/Getting_started ? [11:09:02] yes [11:09:39] but I am faced with "rcsprinter@willow:~$" not "$HOME/public_html/" [11:11:00] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [11:11:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105137.000000 [11:12:04] $HOME is a variable for your home-directory. public-html is a sub-directory inside your home. I guess you need to learn something about unix/linux-shells first. There are plenty of turials in the www. [11:15:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:15:20] Load avg. on willow is WARNING: WARNING - load average: 15.81, 15.27, 15.45 [11:15:22] ok.. [11:16:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:19:20] Load avg. on willow is OK: OK - load average: 13.69, 14.23, 14.96 [11:22:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40550.000000 [11:29:20] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [11:30:51] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:31:00] Rcsprinter: how are you running the bot currently? [11:31:10] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:31:30] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 103286 [11:46:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.074707/1.75, alarm hl:np_load_avg=1.775879/2.0, alarm hl:mem_free=404.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.074707/1.9, alarm hl:np_load_long=1.687988/2.25, alarm hl:mem_free=404.000000M/200M, alarm hl:available=1/0 [11:47:20] Load avg. on willow is WARNING: WARNING - load average: 16.07, 14.39, 13.62 [11:48:17] In a few moments, s1 will be under 100.000 replag :) [11:48:20] Load avg. on willow is OK: OK - load average: 14.04, 14.07, 13.55 [11:49:12] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:49:20] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:20] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:49:21] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:00] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:50:11] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:50:11] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:50:31] now [11:52:57] @replag [11:52:57] valhallasw: s1-rr-a: 1d 3h 42m 28s [-2.57 s/s]; s1-user: 1d 3h 42m 28s [-2.57 s/s]; s3-rr-a: 36s [+0.00 s/s]; s3-user: 36s [+0.00 s/s] [11:53:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.487793/1.75, alarm hl:np_load_avg=1.687500/2.0, alarm hl:mem_free=304.000000M/350M, alarm hl:available=1/0 [11:53:13] gaining fast, too [11:53:22] or rather: catching up fast [12:00:20] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.026367/1.00, alarm hl:np_load_long=0.776367/1.50, alarm hl:mem_free=18748.000000M/350M, alarm hl:available=1/0 [12:01:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:04:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:06:21] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.152344/1.10, alarm hl:np_load_long=0.835938/1.55, alarm hl:mem_free=18679.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.152344/1.00, alarm hl:np_load_long=0.835938/1.50, alarm hl:mem_free=18679.000000M/350M, alarm hl:available=1/0 [12:09:52] 3(resolved) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Hoo man) [12:11:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [12:11:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 95947.000000 [12:12:20] Load avg. on willow is WARNING: WARNING - load average: 16.26, 16.00, 14.62 [12:15:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:16:20] Load avg. on willow is OK: OK - load average: 13.11, 14.89, 14.51 [12:17:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:22:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44153.000000 [12:23:51] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:28:50] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:29:20] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:30:20] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.001953/1.00, alarm hl:np_load_long=0.787110/1.50, alarm hl:mem_free=18993.000000M/350M, alarm hl:available=1/0 [12:30:51] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:31:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:31:20] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:31:30] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 92224 [12:44:31] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.073242/1.10, alarm hl:np_load_long=1.126953/1.55, alarm hl:mem_free=19111.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=3.073242/1.00, alarm hl:np_load_long=1.126953/1.50, alarm hl:mem_free=19111.000000M/350M, alarm hl:available=1/0 [12:47:31] Load avg. on willow is WARNING: WARNING - load average: 15.41, 13.63, 13.23 [12:48:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.842285/1.75, alarm hl:np_load_avg=1.707031/2.0, alarm hl:mem_free=530.000000M/350M, alarm hl:available=1/0 [12:49:32] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:49:32] Load avg. on willow is OK: OK - load average: 10.99, 13.01, 13.07 [12:53:53] @replag [12:53:55] Joan: s1-rr-a: 1d 24m 27s [-3.25 s/s]; s1-user: 1d 24m 27s [-3.25 s/s]; s3-rr-a: 30s [-0.00 s/s]; s3-user: 30s [-0.00 s/s] [13:04:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:11:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [13:12:31] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=8.701172/1.10, alarm hl:np_load_long=1.797851/1.55, alarm hl:mem_free=19487.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=8.701172/1.00, alarm hl:np_load_long=1.797851/1.50, alarm hl:mem_free=19487.000000M/350M, alarm hl:available=1/0 [13:12:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87345.000000 [13:13:32] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 33.03, 17.69, 9.11 [13:13:34] [[Special:Log/newusers]] create 10 * Thomas7 * (New user account) [13:15:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:16:31] Load avg. on ortelius is WARNING: WARNING - load average: 23.24, 22.77, 12.92 [13:17:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:20:32] Load avg. on ortelius is OK: OK - load average: 4.91, 13.39, 11.28 [13:22:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47765.000000 [13:29:33] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [13:31:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:31:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:31:33] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85181 [13:34:32] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:48:31] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.816895/1.75, alarm hl:np_load_avg=1.506348/2.0, alarm hl:mem_free=508.000000M/350M, alarm hl:available=1/0 [13:50:31] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:54:31] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:54:52] 3(commented) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [13:55:33] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:02:32] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:03] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:04:42] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:04:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:05:41] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:07:33] Load avg. on willow is WARNING: WARNING - load average: 15.73, 15.56, 13.96 [14:08:34] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.923828/1.75, alarm hl:np_load_avg=1.933594/2.0, alarm hl:mem_free=173.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.923828/1.9, alarm hl:np_load_long=1.748047/2.25, alarm hl:mem_free=173.000000M/200M, alarm hl:available=1/0 [14:11:02] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [14:12:41] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81796.000000 [14:13:31] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:13:31] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:15:41] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:17:33] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:19:33] Load avg. on willow is OK: OK - load average: 12.07, 14.77, 14.86 [14:19:33] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:22:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51368.000000 [14:25:11] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1958 [14:25:42] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1946 [14:27:32] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.117676/1.75, alarm hl:np_load_avg=1.412109/2.0, alarm hl:mem_free=291.000000M/350M, alarm hl:available=1/0 [14:27:53] 3(commented) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Hoo man) [14:28:11] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1953 [14:29:33] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [14:29:56] 3(commented) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [14:31:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:31:33] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:31:33] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 80051 [14:31:53] 3(commented) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Hoo man) [14:32:33] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:33:54] 3(commented) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Franz Herbach) [14:52:02] Alchimista * Re: [Toolserver-l] UnicodeEncodeError [14:55:11] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [14:55:11] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:42] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:55:51] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:52] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:52] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:55:52] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:56:02] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:56:11] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:56:41] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3610 [14:56:42] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:56:42] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:56:42] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:56:42] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:58:41] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.178711/1.10, alarm hl:np_load_long=0.735352/1.55, alarm hl:mem_free=19639.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.178711/1.00, alarm hl:np_load_long=0.735352/1.50, alarm hl:mem_free=19639.000000M/350M, alarm hl:available=1/0 [14:59:11] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3625 [14:59:42] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:04:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:09:44] Load avg. on willow is WARNING: WARNING - load average: 15.38, 15.04, 14.17 [15:11:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [15:12:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76692.000000 [15:14:41] Load avg. on willow is OK: OK - load average: 13.36, 14.84, 14.48 [15:14:41] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:15:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:17:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:22:31] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3465 [15:22:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54975.000000 [15:29:44] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [15:31:11] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:31:42] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75957 [15:31:42] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:38:43] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [15:39:42] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2635 [15:42:44] Load avg. on willow is WARNING: WARNING - load average: 15.11, 14.83, 14.43 [15:43:44] Load avg. on willow is OK: OK - load average: 13.52, 14.53, 14.36 [15:44:51] MySQL slave on z-dat-s6-a is OK: Uptime: 450184 Threads: 11 Questions: 70527904 Slow queries: 31970 Opens: 595295 Flush tables: 2 Open tables: 1827 Queries per second avg: 156.664 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1737 [15:51:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.551270/1.75, alarm hl:np_load_avg=1.493652/2.0, alarm hl:mem_free=175.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.551270/1.9, alarm hl:np_load_long=1.634766/2.25, alarm hl:mem_free=175.000000M/200M, alarm hl:available=1/0 [15:55:21] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:43] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:55:43] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:55:43] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:55:52] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [15:55:52] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:53] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:53] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:53] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:53] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:00] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:13] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:13] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:13] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:13] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:13] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:13] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:13] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:14] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:14] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:15] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:15] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:16] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:31] /tmp on hyacinth is OK: DISK OK - free space: /tmp 1030 MB (99% inode=99%): [15:56:31] / on hyacinth is OK: DISK OK - free space: / 11595 MB (38% inode=87%): [15:56:44] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:52] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [15:56:52] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [15:56:52] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:52] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:52] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:52] / on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:52] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:53] /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:57:12] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [15:57:12] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [15:57:12] / on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:57:12] /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:57:12] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:57:12] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:57:52] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [15:58:02] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 1017 MB (99% inode=99%): [15:58:02] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 187532 MB (19% inode=99%): [15:58:02] / on z-dat-s3-a is OK: DISK OK - free space: / 11595 MB (38% inode=87%): [15:58:12] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [15:58:12] Load avg. on z-dat-s3-a is OK: OK - load average: 0.30, 1.61, 2.35 [15:58:12] Load avg. on z-dat-s4-a is OK: OK - load average: 0.30, 1.61, 2.34 [15:58:22] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [15:58:22] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 1051 MB (99% inode=99%): [15:58:22] / on z-dat-s6-a is OK: DISK OK - free space: / 11595 MB (38% inode=87%): [15:58:22] Load avg. on z-dat-s7-a is OK: OK - load average: 0.30, 1.56, 2.32 [15:58:22] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 187535 MB (19% inode=99%): [15:58:31] MySQL on z-dat-s3-a is OK: Uptime: 3047300 Threads: 18 Questions: 3297518412 Slow queries: 170777 Opens: 22321802 Flush tables: 1 Open tables: 16383 Queries per second avg: 1082.111 [15:58:31] MySQL slave on z-dat-s4-a is OK: Uptime: 2956188 Threads: 12 Questions: 151836799 Slow queries: 36926 Opens: 66429 Flush tables: 1 Open tables: 590 Queries per second avg: 51.362 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 249 [15:58:31] / on z-dat-s4-a is OK: DISK OK - free space: / 11595 MB (38% inode=87%): [15:58:31] MySQL on z-dat-s4-a is OK: Uptime: 2956188 Threads: 10 Questions: 151836820 Slow queries: 36928 Opens: 66429 Flush tables: 1 Open tables: 590 Queries per second avg: 51.362 [15:58:44] MySQL on z-dat-s6-a is OK: Uptime: 451011 Threads: 15 Questions: 70632128 Slow queries: 32020 Opens: 595336 Flush tables: 2 Open tables: 1828 Queries per second avg: 156.608 [15:58:44] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 245.000000 [15:58:44] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 108309 MB (26% inode=99%): [15:58:44] / on z-dat-s7-a is OK: DISK OK - free space: / 11595 MB (38% inode=87%): [15:58:44] SMF on z-dat-s7-a is OK: OK - all services online [15:58:44] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 1048 MB (99% inode=99%): [15:58:44] SMF on z-dat-s6-a is OK: OK - all services online [15:58:45] NTP on hyacinth is OK: NTP OK: Offset 0.000595 secs [15:58:45] SMTP on z-dat-s4-a is OK: SMTP OK - 0.260 sec. response time [15:58:46] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:58:46] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:58:47] Load avg. on z-dat-s6-a is OK: OK - load average: 1.04, 1.65, 2.34 [15:59:01] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [15:59:11] MySQL on z-dat-s7-a is OK: Uptime: 3479752 Threads: 17 Questions: 758235063 Slow queries: 104179 Opens: 5679110 Flush tables: 1 Open tables: 7108 Queries per second avg: 217.899 [16:02:41] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:04:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:07:30] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3548 [16:07:42] Load avg. on willow is WARNING: WARNING - load average: 15.77, 15.38, 14.31 [16:07:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.894043/1.75, alarm hl:np_load_avg=1.903809/2.0, alarm hl:mem_free=178.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.894043/1.9, alarm hl:np_load_long=1.779297/2.25, alarm hl:mem_free=178.000000M/200M, alarm hl:available=1/0 [16:09:52] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:52] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:52] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:01] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:12] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:10:12] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:10:12] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:31] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [16:10:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 35.40, 21.02, 16.52 [16:10:43] Load avg. on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:10:43] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:10:51] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:11:00] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:11:01] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:01] SMTP on hyacinth is OK: SMTP OK - 0.002 sec. response time [16:11:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [16:11:43] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3591 [16:11:43] Load avg. on willow is WARNING: WARNING - load average: 27.38, 21.32, 16.91 [16:11:52] SMTP on z-dat-s3-a is OK: SMTP OK - 0.012 sec. response time [16:12:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74937.000000 [16:15:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:17:51] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:19:57] 3(created) [ACCAPP-488] Solving links to disambiguation pages in Wikipedia via "Personalized" Crowdsourcing; Account Approval; Blocker New Account <10https://jira.toolserver.org/browse/ACCAPP-488> (Amr Ebaid) [16:21:43] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3412 [16:21:43] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:22:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58577.000000 [16:23:42] Load avg. on willow is OK: OK - load average: 7.39, 12.17, 14.50 [16:28:43] MySQL slave on z-dat-s3-a is OK: Uptime: 3049100 Threads: 22 Questions: 3300231519 Slow queries: 170920 Opens: 22342018 Flush tables: 1 Open tables: 16384 Queries per second avg: 1082.362 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1765 [16:30:42] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [16:31:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:31:43] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 74008 [16:31:43] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:32:42] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.052246/1.75, alarm hl:np_load_avg=1.857910/2.0, alarm hl:mem_free=637.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.052246/1.9, alarm hl:np_load_long=1.815430/2.25, alarm hl:mem_free=637.000000M/200M, alarm hl:available=1/0 [16:33:43] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:48:29] @replag [16:48:30] Betacommand: s1-rr-a: 20h 30m 30s [-1.00 s/s]; s1-user: 20h 30m 30s [-1.00 s/s]; s2-user: 1m 24s [-0.08 s/s]; s3-rr-a: 48s [+0.00 s/s]; s3-user: 48s [+0.00 s/s] [16:49:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.763672/1.75, alarm hl:np_load_avg=1.666992/2.0, alarm hl:mem_free=410.000000M/350M, alarm hl:available=1/0 [16:55:01] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:01] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:02] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:02] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:22] Load avg. on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:55:22] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:51] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:55:52] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:55:52] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:55:53] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:55:53] Load avg. on z-dat-s6-a is OK: OK - load average: 1.61, 2.14, 3.51 [16:56:12] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [17:01:56] 3(created) [TS-1346] SVN write access for UTRS; Toolserver: Subversion; Task <10https://jira.toolserver.org/browse/TS-1346> (Chris Howie) [17:02:51] Load avg. on willow is WARNING: WARNING - load average: 14.05, 15.11, 14.29 [17:03:52] 3(created) [TS-1347] sudo access to "unblock" user; Toolserver: General/Unknown; Task <10https://jira.toolserver.org/browse/TS-1347> (Chris Howie) [17:03:54] Load avg. on willow is OK: OK - load average: 13.16, 14.67, 14.18 [17:04:55] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:11:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [17:12:52] Load avg. on willow is WARNING: WARNING - load average: 16.03, 15.86, 14.88 [17:12:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73903.000000 [17:15:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:18:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:22:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62186.000000 [17:30:52] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [17:31:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:31:53] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 73409 [17:31:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:38:53] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.590820/1.75, alarm hl:np_load_avg=1.454590/2.0, alarm hl:mem_free=319.000000M/350M, alarm hl:available=1/0 [17:59:52] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.476562/1.10, alarm hl:np_load_long=0.927735/1.55, alarm hl:mem_free=19879.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.476562/1.00, alarm hl:np_load_long=0.927735/1.50, alarm hl:mem_free=19879.000000M/350M, alarm hl:available=1/0 [18:04:53] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:05:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:05:01] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:07:56] 3(commented) [DBQ-179] Database Query <10https://jira.toolserver.org/browse/DBQ-179> (Hoo man) [18:11:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.603027/1.75, alarm hl:np_load_avg=1.971680/2.0, alarm hl:mem_free=230.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.603027/1.9, alarm hl:np_load_long=1.776855/2.25, alarm hl:mem_free=230.000000M/200M, alarm hl:available=1/0 [18:11:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [18:12:52] Load avg. on willow is WARNING: WARNING - load average: 15.78, 15.59, 14.35 [18:13:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69486.000000 [18:16:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:16:52] Load avg. on willow is OK: OK - load average: 11.57, 14.59, 14.29 [18:17:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [18:18:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:22:52] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65789.000000 [18:28:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.799805/1.75, alarm hl:np_load_avg=1.717774/2.0, alarm hl:mem_free=457.000000M/350M, alarm hl:available=1/0 [18:30:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [18:32:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:32:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:32:52] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 66729 [18:32:52] Load avg. on willow is WARNING: WARNING - load average: 14.62, 15.20, 14.30 [18:46:02] [[w:en:User:Madman]] * Re: [Toolserver-l] Dementia [19:01:41] I will shutdown s6 for some time now (like announced) [19:02:45] how is the enwiki import coming along? [19:03:02] Platonides * Re: [Toolserver-l] Dementia [19:03:03] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [19:03:46] Firebolt: 1 server is finished since yesterday, the other is import pagelinks at the moment [19:04:18] okay [19:05:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:08:53] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [19:10:52] nighty ~ [19:11:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [19:13:02] Load avg. on willow is WARNING: WARNING - load average: 14.52, 15.42, 14.53 [19:13:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64972.000000 [19:15:02] Load avg. on willow is OK: OK - load average: 12.86, 14.62, 14.34 [19:16:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:18:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:20:53] 3(updated) [SWMTBOT-45] Make SWMTBot recongnize commands based on its current nickname instead of its set nickname in SWMTBot.ini <10https://jira.toolserver.org/browse/SWMTBOT-45> (Daniel Salciccioli) [19:20:54] 3(resolved) [SWMTBOT-45] Make SWMTBot recongnize commands based on its current nickname instead of its set nickname in SWMTBot.ini <10https://jira.toolserver.org/browse/SWMTBOT-45> (Daniel Salciccioli) [19:21:13] MySQL slave on z-dat-s6-a is OK: Uptime: 32 Threads: 5 Questions: 927 Slow queries: 1 Opens: 205 Flush tables: 1 Open tables: 78 Queries per second avg: 28.968 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1227 [19:22:12] ok, finished [19:23:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69400.000000 [19:25:02] DaB. * Re: [Toolserver-announce] [Toolserver-l] Corruption of s6 and (short) downtime of s6 tomorrow (Tuesday) [19:28:30] DaBPunkt, when will the new accounts be added? Within the next week? [19:28:47] today or tomorrow [19:28:55] ah, good. [19:30:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [19:32:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:32:53] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63677 [19:33:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:34:02] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:45:02] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.028320/1.00, alarm hl:np_load_long=0.877930/1.50, alarm hl:mem_free=19298.000000M/350M, alarm hl:available=1/0 [19:46:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:01:05] @replag [20:01:06] DaBPunkt: s1-rr-a: 16h 57m 8s [-1.11 s/s]; s1-user: 16h 57m 8s [-1.11 s/s]; s3-rr-a: 38s [-0.00 s/s]; s3-user: 38s [-0.00 s/s] [20:02:20] :o not several weeks this time [20:03:03] Load avg. on willow is WARNING: WARNING - load average: 13.95, 15.30, 13.80 [20:04:03] Load avg. on willow is OK: OK - load average: 13.61, 14.97, 13.78 [20:05:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:07:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.765625/1.75, alarm hl:np_load_avg=1.894043/2.0, alarm hl:mem_free=778.000000M/350M, alarm hl:available=1/0 [20:08:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:10:01] Sactage: it was never "several weeks" (at least not this time), but yeah [20:10:30] I thought I saw lag of 6w last night [20:10:34] might have misread [20:11:02] s1-rr-a: 2d 22h 31m 11s [-3.37 s/s]; s1-user: 2w 1d 1h 42m 24s [+1.00 s/s]; s2-user: 1h 43m 46s [-0.20 s/s]; s2-user-c: 2h 57m 38s [-0.07 s/s]; s5-user-c: 2h 57m 38s [-0.07 s/s] [20:11:08] oh, 2w [20:11:14] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [20:13:02] Load avg. on willow is WARNING: WARNING - load average: 16.98, 15.66, 14.59 [20:13:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59802.000000 [20:13:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.134277/1.75, alarm hl:np_load_avg=1.954590/2.0, alarm hl:mem_free=445.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.134277/1.9, alarm hl:np_load_long=1.820801/2.25, alarm hl:mem_free=445.000000M/200M, alarm hl:available=1/0 [20:14:01] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [20:16:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:17:12] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.416016/1.10, alarm hl:np_load_long=0.970703/1.55, alarm hl:mem_free=19645.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.416016/1.00, alarm hl:np_load_long=0.970703/1.50, alarm hl:mem_free=19645.000000M/350M, alarm hl:available=1/0 [20:18:13] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:18:22] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:23:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73002.000000 [20:30:40] @replag [20:30:41] JeffQuassel_: s1-rr-a: 15h 45m 18s [-2.43 s/s]; s1-user: 15h 45m 18s [-2.43 s/s]; s2-user: 18s [-0.00 s/s]; s3-rr-a: 17s [-0.01 s/s]; s3-user: 17s [-0.01 s/s] [20:30:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:32:22] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:32:54] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 56331 [20:33:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:06:26] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:11:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [21:14:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47080.000000 [21:16:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:18:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:23:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76602.000000 [21:30:52] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:32:02] Load avg. on willow is WARNING: WARNING - load average: 13.21, 15.50, 14.11 [21:32:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.750976/1.75, alarm hl:np_load_avg=1.964355/2.0, alarm hl:mem_free=456.000000M/350M, alarm hl:available=1/0 [21:32:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:32:54] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43400 [21:33:02] Load avg. on willow is OK: OK - load average: 9.62, 14.09, 13.70 [21:33:23] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:34:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:43:23] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.781738/1.75, alarm hl:np_load_avg=1.753418/2.0, alarm hl:mem_free=370.000000M/350M, alarm hl:available=1/0 [21:44:23] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.094726/1.00, alarm hl:np_load_long=0.804688/1.50, alarm hl:mem_free=19328.000000M/350M, alarm hl:available=1/0 [21:45:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:58:53] 3(resolved) [OSM-10] mapnik doesn't run from console <10https://jira.toolserver.org/browse/OSM-10> (Kolossos ) [22:00:14] @replag [22:00:15] Joan: s1-rr-a: 10h 57m 43s [-3.21 s/s]; s1-user: 10h 57m 43s [-3.21 s/s]; s2-user: 12s [-0.00 s/s]; s3-rr-a: 39s [+0.00 s/s]; s3-user: 39s [+0.00 s/s] [22:00:28] \o/ [22:00:31] * Joan hugs DaBPunkt. [22:00:36] Thanks for all your work on this. :-) [22:00:47] np [22:02:54] [[Database access]] ! 10https://wiki.toolserver.org/w/index.php?diff=6993&oldid=6983&rcid=9209 * Tb * (+20) (Replace with to improve rendering) [22:03:02] Load avg. on willow is WARNING: WARNING - load average: 15.73, 15.53, 14.57 [22:03:59] [[Database access]] ! 10https://wiki.toolserver.org/w/index.php?diff=6994&oldid=6993&rcid=9210 * Tb * (+25) (/* language */ ) [22:07:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:10:02] Load avg. on willow is OK: OK - load average: 13.71, 14.84, 14.67 [22:12:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [22:14:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38213.000000 [22:15:49] http://munin.toolserver.org/Database/thyme/mysql_replication.html – nice ski-mountain ;) [22:16:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:18:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:18:54] 3(updated) [TS-1333] Install ack <10https://jira.toolserver.org/browse/TS-1333> (Krinkle) [22:19:45] FMA? Full Metal Alchemist on yarrow? [22:22:11] nacht ts [22:23:02] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80203.000000 [22:25:38] @replag [22:25:39] Dispenser: s1-rr-a: 10h 19m 45s [-1.50 s/s]; s1-user: 10h 19m 45s [-1.50 s/s]; s2-user: 17s [+0.00 s/s]; s3-rr-a: 59s [+0.01 s/s]; s3-user: 59s [+0.01 s/s] [22:30:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [22:32:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:33:53] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36653 [22:34:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:51:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.866699/1.75, alarm hl:np_load_avg=1.889160/2.0, alarm hl:mem_free=422.000000M/350M, alarm hl:available=1/0 [22:52:11] Load avg. on willow is WARNING: WARNING - load average: 15.04, 15.14, 14.49 [22:52:42] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [22:53:12] Load avg. on willow is OK: OK - load average: 13.59, 14.70, 14.37 [22:56:43] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.635254/1.75, alarm hl:np_load_avg=1.780762/2.0, alarm hl:mem_free=333.000000M/350M, alarm hl:available=1/0 [22:59:52] 3(created) [UTRS-96] HTTP 404 makes a rather concerning privacy policy; UTRS: Main Interface; Bug <10https://jira.toolserver.org/browse/UTRS-96> (Simon Walker) [23:02:12] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:03:11] Load avg. on willow is WARNING: WARNING - load average: 18.30, 17.05, 15.50 [23:07:32] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:12:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Unknown database enwiki_p [23:14:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34114.000000 [23:16:42] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:19:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:19:12] Load avg. on willow is OK: OK - load average: 11.88, 14.05, 14.91 [23:23:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83805.000000 [23:31:02] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [23:31:12] Load avg. on willow is WARNING: WARNING - load average: 21.24, 17.17, 15.45 [23:32:32] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:34:12] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32987 [23:34:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:35:14] Load avg. on willow is OK: OK - load average: 14.37, 14.95, 14.89 [23:48:53] 3(created) [ACCAPP-489] I'd like to help develop and run a bot which will help the Teahouse project on en.wikipedia; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-489> (Andrew Skoda) [23:56:54] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.028809/1.75, alarm hl:np_load_avg=1.771973/2.0, alarm hl:mem_free=263.000000M/350M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.028809/1.9, alarm hl:np_load_long=1.757324/2.25, alarm hl:mem_free=263.000000M/200M, alarm hl:available=1/0