[00:13:24] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.083008/1.8, alarm hl:np_load_avg=1.083496/2.3, alarm hl:mem_free=125.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.083008/1.9, alarm hl:np_load_long=0.976562/2.25, alarm hl:mem_free=125.000000M/200M, alarm hl:available=1/0 [00:15:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10833.000000 [00:16:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10861.000000 [00:23:25] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [00:29:54] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 274 bytes in 11.168 second response time [00:32:02] [[w:en:User:Madman]] * Re: [Toolserver-l] [Wikitech-l] 403: User account expired toolserver.org/~soxred93 [00:42:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [00:43:35] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.711914/1.8, alarm hl:np_load_avg=0.698242/2.3, alarm hl:mem_free=224.000000M/300M, alarm hl:available=1/0 [00:44:44] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:44:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:45:34] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.068359/1.00, alarm hl:np_load_long=0.871094/1.50, alarm hl:mem_free=20726.000000M/300M, alarm hl:available=1/0 [00:45:34] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:45:44] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:46:34] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [00:48:13] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 473252 MB (8% inode=45%): [00:49:34] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:02:33] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.801758/1.8, alarm hl:np_load_avg=0.710449/2.3, alarm hl:mem_free=158.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.801758/1.9, alarm hl:np_load_long=0.713379/2.25, alarm hl:mem_free=158.000000M/200M, alarm hl:available=1/0 [01:07:14] fisheye.toolserver.org on web.amaranth is CRITICAL: CRITICAL - Socket timeout after 21 seconds [01:13:34] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [01:15:44] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14437.000000 [01:16:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14462.000000 [01:30:03] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 272 bytes in 10.517 second response time [01:40:44] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.717773/1.8, alarm hl:np_load_avg=0.648438/2.3, alarm hl:mem_free=299.000000M/300M, alarm hl:available=1/0 [01:42:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [01:44:46] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:45:05] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:45:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:45:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:48:14] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 472744 MB (8% inode=45%): [01:51:44] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:02:45] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.673340/1.8, alarm hl:np_load_avg=0.671387/2.3, alarm hl:mem_free=238.000000M/300M, alarm hl:available=1/0 [02:07:45] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [02:09:52] @replag [02:09:53] Joan: s1-rr-a: 4h 54m 49s [+1.00 s/s]; s1-user: 4h 54m 49s [+1.00 s/s]; s3-rr-a: 16s [-0.00 s/s]; s3-user: 16s [-0.00 s/s]; s5-rr-a: 12s [+0.00 s/s]; s5-user: 12s [+0.00 s/s] [02:10:34] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.139649/1.10, alarm hl:np_load_long=0.853515/1.55, alarm hl:mem_free=21705.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.139649/1.00, alarm hl:np_load_long=0.853515/1.50, alarm hl:mem_free=21705.000000M/300M, alarm hl:available=1/0 [02:11:33] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:15:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18037.000000 [02:16:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18063.000000 [02:33:53] 3(created) [UTRS-80] Typos in user permission IRC notifications; UTRS: Main Interface; Bug <10https://jira.toolserver.org/browse/UTRS-80> (Hersfold) [02:42:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [02:44:34] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.617188/1.10, alarm hl:np_load_long=1.054688/1.55, alarm hl:mem_free=21829.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.617188/1.00, alarm hl:np_load_long=1.054688/1.50, alarm hl:mem_free=21829.000000M/300M, alarm hl:available=1/0 [02:44:47] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:45:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:45:47] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:45:54] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:46:52] @replag [02:46:53] Joan: s1-rr-a: 5h 31m 48s [+1.00 s/s]; s1-user: 5h 31m 48s [+1.00 s/s]; s2-user: 42s [+0.00 s/s]; s3-rr-a: 50s [+0.02 s/s]; s3-user: 51s [+0.02 s/s]; s6-rr-a: 23s [+0.00 s/s]; s6-user: 23s [+0.00 s/s] [02:47:33] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [02:49:14] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 470290 MB (8% inode=44%): [02:59:53] 3(commented) [UTRS-78] Many links have interfering JavaScript <10https://jira.toolserver.org/browse/UTRS-78> (Hersfold) [03:12:52] 3(created) [UTRS-82] Add blacklist to email field on appeal form; UTRS: Main Interface; New Feature <10https://jira.toolserver.org/browse/UTRS-82> (Hersfold) [03:15:07] @replag [03:15:07] matthewrbowker: s1-rr-a: 6h 3s [+1.00 s/s]; s1-user: 6h 3s [+1.00 s/s]; s2-user: 14s [-0.02 s/s]; s3-rr-a: 7m 41s [+0.24 s/s]; s3-user: 7m 41s [+0.24 s/s]; s6-rr-a: 13s [-0.01 s/s]; s6-user: 13s [-0.01 s/s] [03:15:21] * matthewrbowker sincerely hopes just replag is broken :/ [03:15:50] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21640.000000 [03:16:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21678.000000 [03:26:07] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:37:29] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.051758/1.00, alarm hl:np_load_long=0.810547/1.50, alarm hl:mem_free=21406.000000M/300M, alarm hl:available=1/0 [03:38:37] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [03:41:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.740723/1.8, alarm hl:np_load_avg=0.725586/2.3, alarm hl:mem_free=242.000000M/300M, alarm hl:available=1/0 [03:42:40] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.192383/1.10, alarm hl:np_load_long=0.906250/1.55, alarm hl:mem_free=21014.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.192383/1.00, alarm hl:np_load_long=0.906250/1.50, alarm hl:mem_free=21014.000000M/300M, alarm hl:available=1/0 [03:42:57] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [03:44:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:45:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:45:48] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:45:58] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:48:49] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [03:49:28] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 469989 MB (8% inode=44%): [04:10:47] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [04:13:48] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.756348/1.8, alarm hl:np_load_avg=0.797363/2.3, alarm hl:mem_free=190.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.756348/1.9, alarm hl:np_load_long=0.772461/2.25, alarm hl:mem_free=190.000000M/200M, alarm hl:available=1/0 [04:14:48] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [04:15:57] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25250.000000 [04:17:27] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25338.000000 [04:17:46] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.573242/1.8, alarm hl:np_load_avg=0.704102/2.3, alarm hl:mem_free=160.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.573242/1.9, alarm hl:np_load_long=0.740234/2.25, alarm hl:mem_free=160.000000M/200M, alarm hl:available=1/0 [04:27:53] 3(created) [UTRS-83] Comments cut off in appeals; UTRS; Bug <10https://jira.toolserver.org/browse/UTRS-83> (DeltaQuad) [04:29:53] 3(commented) [UTRS-83] Comments cut off in appeals <10https://jira.toolserver.org/browse/UTRS-83> (DeltaQuad) [04:31:54] 3(commented) [UTRS-79] Sent email content in an appeal's log discloses the appealing user's email address <10https://jira.toolserver.org/browse/UTRS-79> (DeltaQuad) [04:43:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [04:45:13] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:45:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:46:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:46:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:49:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 469286 MB (8% inode=44%): [05:02:59] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.735352/1.8, alarm hl:np_load_avg=0.705078/2.3, alarm hl:mem_free=206.000000M/300M, alarm hl:available=1/0 [05:06:00] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [05:13:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.824219/1.8, alarm hl:np_load_avg=0.766113/2.3, alarm hl:mem_free=290.000000M/300M, alarm hl:available=1/0 [05:16:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28856.000000 [05:17:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28956.000000 [05:22:53] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.457031/1.10, alarm hl:np_load_long=0.872070/1.55, alarm hl:mem_free=21042.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.457031/1.00, alarm hl:np_load_long=0.872070/1.50, alarm hl:mem_free=21042.000000M/300M, alarm hl:available=1/0 [05:27:51] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [05:44:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [05:45:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:46:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:46:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:46:12] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:49:53] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 468773 MB (8% inode=44%): [06:03:12] Load avg. on willow is WARNING: WARNING - load average: 15.19, 18.66, 13.97 [06:03:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=2.013672/1.8, alarm hl:np_load_avg=2.367188/2.3, alarm hl:mem_free=294.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=2.013672/1.9, alarm hl:np_load_long=1.751465/2.25, alarm hl:mem_free=294.000000M/200M, alarm hl:available=1/0 [06:04:11] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [06:07:12] Load avg. on willow is OK: OK - load average: 8.84, 13.93, 13.09 [06:08:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.366699/1.8, alarm hl:np_load_avg=1.711914/2.3, alarm hl:mem_free=219.000000M/300M, alarm hl:available=1/0 [06:16:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32455.000000 [06:18:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32615.000000 [06:45:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [06:45:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:46:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:46:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:47:10] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:50:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 468271 MB (8% inode=44%): [07:01:10] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:10:59] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:16:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36056.000000 [07:18:50] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36216.000000 [07:43:59] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.564453/1.10, alarm hl:np_load_long=0.931640/1.55, alarm hl:mem_free=21724.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.564453/1.00, alarm hl:np_load_long=0.931640/1.50, alarm hl:mem_free=21724.000000M/300M, alarm hl:available=1/0 [07:45:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [07:45:29] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:45:59] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [07:46:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:46:11] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:47:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:50:52] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467771 MB (8% inode=44%): [07:53:55] 3(commented) [ACCAPP-105] To Run"Addbot" and to create and host various small web scripts <10https://jira.toolserver.org/browse/ACCAPP-105> (Marlen Caemmerer) [08:08:20] @replag [08:08:21] maityankit: s1-rr-a: 10h 53m 16s [+1.00 s/s]; s1-user: 10h 53m 16s [+1.00 s/s]; s3-rr-a: 43s [-0.02 s/s]; s3-user: 43s [-0.02 s/s] [08:10:02] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.190430/1.10, alarm hl:np_load_long=0.940430/1.55, alarm hl:mem_free=21245.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.190430/1.00, alarm hl:np_load_long=0.940430/1.50, alarm hl:mem_free=21245.000000M/300M, alarm hl:available=1/0 [08:11:01] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:12:10] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.134277/1.8, alarm hl:np_load_avg=1.173828/2.3, alarm hl:mem_free=177.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.134277/1.9, alarm hl:np_load_long=1.147461/2.25, alarm hl:mem_free=177.000000M/200M, alarm hl:available=1/0 [08:14:10] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [08:16:10] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39658.000000 [08:19:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39826.000000 [08:45:01] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.227539/1.10, alarm hl:np_load_long=0.919922/1.55, alarm hl:mem_free=20797.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.227539/1.00, alarm hl:np_load_long=0.919922/1.50, alarm hl:mem_free=20797.000000M/300M, alarm hl:available=1/0 [08:45:20] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [08:45:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:46:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [08:46:20] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:46:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:46:20] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:48:20] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:51:01] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 467191 MB (8% inode=44%): [08:55:48] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:13:19] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.619629/1.8, alarm hl:np_load_avg=1.528320/2.3, alarm hl:mem_free=233.000000M/300M, alarm hl:available=1/0 [09:16:20] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [09:17:10] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43318.000000 [09:20:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43491.000000 [09:45:11] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.307617/1.10, alarm hl:np_load_long=0.897461/1.55, alarm hl:mem_free=20794.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.307617/1.00, alarm hl:np_load_long=0.897461/1.50, alarm hl:mem_free=20794.000000M/300M, alarm hl:available=1/0 [09:45:30] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [09:45:40] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:46:31] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:46:31] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:47:10] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [09:48:31] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:51:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 466669 MB (8% inode=44%): [09:52:52] 3(updated) [ACCAPP-480] Account Request TU Vienna, Student member (PhD) <10https://jira.toolserver.org/browse/ACCAPP-480> (Thomas Leber) [09:52:54] 3(updated) [ACCAPP-480] Account Request TU Vienna, Student member (PhD) <10https://jira.toolserver.org/browse/ACCAPP-480> (Thomas Leber) [09:52:55] 3(updated) [ACCAPP-480] Account Request TU Vienna, Student member (PhD) <10https://jira.toolserver.org/browse/ACCAPP-480> (Thomas Leber) [09:58:55] 3(commented) [ACCAPP-480] Account Request TU Vienna, Student member (PhD) <10https://jira.toolserver.org/browse/ACCAPP-480> (Thomas Leber) [10:17:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46920.000000 [10:17:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.113770/1.8, alarm hl:np_load_avg=1.148438/2.3, alarm hl:mem_free=286.000000M/300M, alarm hl:available=1/0 [10:19:13] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.357422/1.10, alarm hl:np_load_long=0.884765/1.55, alarm hl:mem_free=19596.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.357422/1.00, alarm hl:np_load_long=0.884765/1.50, alarm hl:mem_free=19596.000000M/300M, alarm hl:available=1/0 [10:20:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47095.000000 [10:20:30] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [10:22:14] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [10:26:12] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.576172/1.10, alarm hl:np_load_long=1.006836/1.55, alarm hl:mem_free=18829.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.576172/1.00, alarm hl:np_load_long=1.006836/1.50, alarm hl:mem_free=18829.000000M/300M, alarm hl:available=1/0 [10:40:32] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [10:45:41] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [10:45:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:46:39] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:46:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:48:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:51:10] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 466120 MB (8% inode=44%): [10:55:42] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:42] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:31] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:56:31] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:57:51] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.574219/1.8, alarm hl:np_load_avg=1.534668/2.3, alarm hl:mem_free=255.000000M/300M, alarm hl:available=1/0 [10:58:51] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [11:02:31] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.485351/1.10, alarm hl:np_load_long=1.412109/1.55, alarm hl:mem_free=18677.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.485351/1.00, alarm hl:np_load_long=1.412109/1.50, alarm hl:mem_free=18677.000000M/300M, alarm hl:available=1/0 [11:11:50] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.800781/1.8, alarm hl:np_load_avg=1.718262/2.3, alarm hl:mem_free=223.000000M/300M, alarm hl:available=1/0 [11:14:40] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:40] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:52] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:14:59] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:59] / on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:59] /sql on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:59] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:59] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:59] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:14:59] Load avg. on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:15:09] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:15:19] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:20] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:20] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:20] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:20] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:29] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:15:30] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:15:39] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:15:40] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:16:01] / on z-dat-s4-a is OK: DISK OK - free space: / 11635 MB (38% inode=87%): [11:16:02] Load avg. on z-dat-s4-a is OK: OK - load average: 0.11, 1.18, 1.60 [11:16:02] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 113625 MB (28% inode=99%): [11:16:02] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [11:16:10] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [11:16:10] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [11:16:10] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 2865 MB (99% inode=99%): [11:16:19] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [11:16:19] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [11:16:19] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:29] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [11:16:39] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [11:16:39] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [11:16:49] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [11:17:20] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50531.000000 [11:19:00] SMF on z-dat-s7-a is OK: OK - all services online [11:19:00] MySQL on z-dat-s7-a is OK: Uptime: 2166941 Threads: 13 Questions: 470487089 Slow queries: 73811 Opens: 3409144 Flush tables: 1 Open tables: 6913 Queries per second avg: 217.120 [11:19:00] SMF on hyacinth is OK: OK - all services online [11:19:01] SMF on z-dat-s4-a is OK: OK - all services online [11:19:01] SMF on z-dat-s3-a is OK: OK - all services online [11:19:01] SMF on z-dat-s6-a is OK: OK - all services online [11:19:01] MySQL slave on z-dat-s4-a is OK: Uptime: 1643409 Threads: 11 Questions: 69183233 Slow queries: 23176 Opens: 34755 Flush tables: 1 Open tables: 509 Queries per second avg: 42.97 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 386 [11:19:01] MySQL on z-dat-s4-a is OK: Uptime: 1643409 Threads: 11 Questions: 69183233 Slow queries: 23176 Opens: 34755 Flush tables: 1 Open tables: 509 Queries per second avg: 42.97 [11:19:01] MySQL slave on z-dat-s3-a is OK: Uptime: 1734524 Threads: 17 Questions: 1937082954 Slow queries: 118059 Opens: 13364630 Flush tables: 1 Open tables: 16384 Queries per second avg: 1116.780 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 395 [11:19:02] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:19:02] MySQL on z-dat-s3-a is OK: Uptime: 1734527 Threads: 17 Questions: 1937085092 Slow queries: 118059 Opens: 13364634 Flush tables: 1 Open tables: 16384 Queries per second avg: 1116.780 [11:19:03] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 354.000000 [11:19:10] MySQL on z-dat-s6-a is OK: Uptime: 1643421 Threads: 7 Questions: 251108445 Slow queries: 111187 Opens: 2751474 Flush tables: 2 Open tables: 2749 Queries per second avg: 152.796 [11:19:10] MySQL slave on z-dat-s6-a is OK: Uptime: 1643421 Threads: 7 Questions: 251108454 Slow queries: 111187 Opens: 2751474 Flush tables: 2 Open tables: 2749 Queries per second avg: 152.796 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 369 [11:19:10] SMTP on z-dat-s3-a is OK: SMTP OK - 0.004 sec. response time [11:19:10] SMTP on hyacinth is OK: SMTP OK - 0.003 sec. response time [11:19:10] SMTP on z-dat-s4-a is OK: SMTP OK - 0.055 sec. response time [11:19:10] NTP on hyacinth is OK: NTP OK: Offset 0.000201 secs [11:19:10] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:19:11] SMTP on z-dat-s6-a is OK: SMTP OK - 0.040 sec. response time [11:19:19] MySQL slave on z-dat-s7-a is OK: Uptime: 2166962 Threads: 11 Questions: 470491354 Slow queries: 73813 Opens: 3409146 Flush tables: 1 Open tables: 6913 Queries per second avg: 217.120 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 338 [11:19:20] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [11:19:20] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:19:40] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:19:40] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:19:40] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:20:10] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50701.000000 [11:25:52] 3(commented) [ACCAPP-481] I am active user in http://fa.wiktionary.org and i want this account to run bot on this wiki also updating it's statics. <10https://jira.toolserver.org/browse/ACCAPP-481> (Abdollah Abdollahi) [11:29:54] 3(commented) [ACCAPP-481] I am active user in http://fa.wiktionary.org and i want this account to run bot on this wiki also updating it's statics. <10https://jira.toolserver.org/browse/ACCAPP-481> (Abdollah Abdollahi) [11:33:39] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:33:54] 3(commented) [ACCAPP-481] I am active user in http://fa.wiktionary.org and i want this account to run bot on this wiki also updating it's statics. <10https://jira.toolserver.org/browse/ACCAPP-481> (Abdollah Abdollahi) [11:36:42] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.363281/1.10, alarm hl:np_load_long=1.331055/1.55, alarm hl:mem_free=18460.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.363281/1.00, alarm hl:np_load_long=1.331055/1.50, alarm hl:mem_free=18460.000000M/300M, alarm hl:available=1/0 [11:45:53] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [11:46:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:46:56] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:46:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:49:01] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:51:22] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 465597 MB (8% inode=44%): [11:59:52] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:02:53] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.896484/1.10, alarm hl:np_load_long=1.331055/1.55, alarm hl:mem_free=19177.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.896484/1.00, alarm hl:np_load_long=1.331055/1.50, alarm hl:mem_free=19177.000000M/300M, alarm hl:available=1/0 [12:13:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.616699/1.8, alarm hl:np_load_avg=1.546875/2.3, alarm hl:mem_free=217.000000M/300M, alarm hl:available=1/0 [12:14:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [12:17:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54133.000000 [12:17:51] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [12:20:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54309.000000 [12:46:01] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [12:46:42] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:47:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:47:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:49:02] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.252930/1.10, alarm hl:np_load_long=1.337891/1.55, alarm hl:mem_free=18756.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.252930/1.00, alarm hl:np_load_long=1.337891/1.50, alarm hl:mem_free=18756.000000M/300M, alarm hl:available=1/0 [12:49:11] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:51:31] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 465095 MB (8% inode=44%): [13:03:03] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.527344/1.8, alarm hl:np_load_avg=1.520020/2.3, alarm hl:mem_free=259.000000M/300M, alarm hl:available=1/0 [13:04:02] Sun Grid Engine execd on wolfsbane is WARNING: short-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.787109/1.10, alarm hl:np_load_long=0.841309/1.55, alarm hl:mem_free=148.000000M/300M, alarm hl:available=1/0: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.787109/1.00, alarm hl:np_load_long=0.841309/1.50, alarm hl:mem_free=148.000000M/300M, alarm hl:available=1/0 [13:05:02] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [13:07:53] 3(commented) [UTRS-78] Many links have interfering JavaScript <10https://jira.toolserver.org/browse/UTRS-78> (Chris Howie) [13:09:02] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:17:48] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57753.000000 [13:20:13] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:20:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57931.000000 [13:23:13] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.721680/1.10, alarm hl:np_load_long=1.346680/1.55, alarm hl:mem_free=18892.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.721680/1.00, alarm hl:np_load_long=1.346680/1.50, alarm hl:mem_free=18892.000000M/300M, alarm hl:available=1/0 [13:39:29] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.443848/1.8, alarm hl:np_load_avg=1.427734/2.3, alarm hl:mem_free=240.000000M/300M, alarm hl:available=1/0 [13:41:28] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [13:46:29] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [13:47:29] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:47:29] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:47:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:49:29] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [13:49:39] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:52:00] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 464534 MB (8% inode=44%): [13:52:28] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.667969/1.10, alarm hl:np_load_long=1.487305/1.55, alarm hl:mem_free=18541.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.667969/1.00, alarm hl:np_load_long=1.487305/1.50, alarm hl:mem_free=18541.000000M/300M, alarm hl:available=1/0 [14:02:30] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.782226/1.8, alarm hl:np_load_avg=1.601562/2.3, alarm hl:mem_free=250.000000M/300M, alarm hl:available=1/0 [14:14:57] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [14:17:58] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.372559/1.8, alarm hl:np_load_avg=1.436524/2.3, alarm hl:mem_free=221.000000M/300M, alarm hl:available=1/0 [14:18:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61376.000000 [14:18:56] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:19:57] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.037109/1.00, alarm hl:np_load_long=0.921387/1.50, alarm hl:mem_free=869.000000M/300M, alarm hl:available=1/0 [14:20:57] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [14:21:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61553.000000 [14:22:00] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.321289/1.10, alarm hl:np_load_long=1.426758/1.55, alarm hl:mem_free=18792.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.321289/1.00, alarm hl:np_load_long=1.426758/1.50, alarm hl:mem_free=18792.000000M/300M, alarm hl:available=1/0 [14:38:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk FTOL, far1-n1-fast2 FTOL, far1-n1-fast CRIT [14:40:57] 3(commented) [MNT-1199] Configured damiana's and tunera's MGMT-IPs and put them into DNS <10https://jira.toolserver.org/browse/MNT-1199> (Marlen Caemmerer) [14:41:34] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:42:05] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:43:19] hi Danny_B|backup, are you here? [14:43:56] 3(commented) [MNT-1199] Configured damiana's and tunera's MGMT-IPs and put them into DNS <10https://jira.toolserver.org/browse/MNT-1199> (Marlen Caemmerer) [14:45:05] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.668945/1.10, alarm hl:np_load_long=1.364258/1.55, alarm hl:mem_free=18367.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.668945/1.00, alarm hl:np_load_long=1.364258/1.50, alarm hl:mem_free=18367.000000M/300M, alarm hl:available=1/0 [14:47:06] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:47:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [14:48:06] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:48:06] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:48:19] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:50:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:52:16] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 463588 MB (8% inode=44%): [15:09:18] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.541992/1.10, alarm hl:np_load_long=1.548828/1.55, alarm hl:mem_free=18955.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.541992/1.00, alarm hl:np_load_long=1.548828/1.50, alarm hl:mem_free=18955.000000M/300M, alarm hl:available=1/0 [15:09:22] alkamid: yup? [15:10:10] Danny_B|backup, for my purposes, I always need the latest and the second to latest dumps in .xml format (counting stats for pl.wiktionary). What should I do if I need to unpack the other .bz2 file from /store? do it in tmp? [15:10:24] or could I request write access to /store/plwiktionary? [15:11:05] or, if you are maintaining the folders with a script, perhaps you could change it so that there are always two last dumps unpacked? [15:12:53] alkamid: what script you use? i mean language? [15:13:05] Danny_B|backup, python / pywikipediabot [15:13:11] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.706055/1.8, alarm hl:np_load_avg=1.648438/2.3, alarm hl:mem_free=171.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.706055/1.9, alarm hl:np_load_long=1.476074/2.25, alarm hl:mem_free=171.000000M/200M, alarm hl:available=1/0 [15:13:32] alkamid: i've been told pywiki can work with .xml.bz2 directly [15:13:57] Danny_B|backup, good point. I'll check it [15:18:31] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65002.000000 [15:21:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65160.000000 [15:21:53] 3(created) [UTRS-84] New form field for blockee to fill: "Why were you blocked?"; UTRS; New Feature <10https://jira.toolserver.org/browse/UTRS-84> (Fluffernutter) [15:26:10] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:26:20] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [15:26:50] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38593 MB (9% inode=99%): [15:31:50] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23470 MB (5% inode=99%): [15:39:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.571777/1.8, alarm hl:np_load_avg=1.622070/2.3, alarm hl:mem_free=245.000000M/300M, alarm hl:available=1/0 [15:46:13] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:47:13] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [15:47:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [15:47:52] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 30884 MB (7% inode=99%): [15:48:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:48:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:48:53] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:48:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 71142 MB (17% inode=99%): [15:49:13] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.208008/1.10, alarm hl:np_load_long=1.318359/1.55, alarm hl:mem_free=18507.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.208008/1.00, alarm hl:np_load_long=1.318359/1.50, alarm hl:mem_free=18507.000000M/300M, alarm hl:available=1/0 [15:50:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:50:45] Danny_B|backup, are you going to take a look at pagecounts stats as well? right now it's a mess, I signalled it in this post: http://lists.wikimedia.org/pipermail/toolserver-l/2012-February/004714.html [15:52:33] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 462885 MB (8% inode=44%): [15:56:02] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:15] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:56:15] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [15:56:23] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:23] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:23] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:42] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [15:56:52] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:56:54] 3(commented) [MNT-1199] Configured damiana's and tunera's MGMT-IPs and put them into DNS <10https://jira.toolserver.org/browse/MNT-1199> (Marlen Caemmerer) [15:57:12] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:57:13] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [15:57:13] SMTP on z-dat-s6-a is OK: SMTP OK - 0.040 sec. response time [16:02:53] Load avg. on willow is WARNING: WARNING - load average: 13.37, 15.08, 14.49 [16:03:52] Load avg. on willow is OK: OK - load average: 11.67, 14.28, 14.24 [16:08:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk FTOL, far1-n1-fast2 FTOL, far1-n1-fast CRIT [16:18:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68604.000000 [16:21:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68762.000000 [16:39:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.716797/1.8, alarm hl:np_load_avg=0.726562/2.3, alarm hl:mem_free=224.000000M/300M, alarm hl:available=1/0 [16:41:13] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [16:44:53] 3(resolved) [MNT-1217] Hard disk failure in SAN <10https://jira.toolserver.org/browse/MNT-1217> (Marlen Caemmerer) [16:46:13] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.899414/1.8, alarm hl:np_load_avg=0.787109/2.3, alarm hl:mem_free=266.000000M/300M, alarm hl:available=1/0 [16:47:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [16:48:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:48:23] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:49:52] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:51:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:52:40] [[Admin:Far1-n1-oe16-esams]] 10https://wiki.toolserver.org/w/index.php?diff=6910&oldid=6395&rcid=9100 * Nosy * (+390) () [16:53:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 462303 MB (8% inode=44%): [16:55:00] [[Special:Log/newusers]] create 10 * Dipankar Chetia * (New user account) [17:06:55] 3(commented) [ACCAPP-481] I am active user in http://fa.wiktionary.org and i want this account to run bot on this wiki also updating it's statics. <10https://jira.toolserver.org/browse/ACCAPP-481> (Darafsh Kaviyani) [17:08:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk FTOL, far1-n1-fast2 FTOL, far1-n1-fast CRIT [17:18:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72203.000000 [17:18:57] 3(commented) [OSM-9] Periodic "Connection refused" on tirex mod_tile socket <10https://jira.toolserver.org/browse/OSM-9> (Marlen Caemmerer) [17:21:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72362.000000 [17:24:33] @replag [17:24:33] Alpha_Quadrant: s1-rr-a: 20h 9m 28s [+1.00 s/s]; s1-user: 20h 9m 28s [+1.00 s/s]; s6-rr-a: 10s [+0.00 s/s]; s6-user: 10s [+0.00 s/s] [17:25:09] why is the replag on s1 so high? [17:31:53] 3(created) [UTRS-85] "void()" is not valid JavaScript; UTRS: Main Interface; Minor Bug <10https://jira.toolserver.org/browse/UTRS-85> (Chris Howie) [17:37:12] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.865234/1.8, alarm hl:np_load_avg=0.854492/2.3, alarm hl:mem_free=179.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.865234/1.9, alarm hl:np_load_long=0.840332/2.25, alarm hl:mem_free=179.000000M/200M, alarm hl:available=1/0 [17:39:54] 3(created) [UTRS-86] The double-quote character is not HTML-encoded in JavaScript HTML attributes; UTRS: Main Interface; Bug <10https://jira.toolserver.org/browse/UTRS-86> (Chris Howie) [17:44:13] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.307617/1.10, alarm hl:np_load_long=0.868164/1.55, alarm hl:mem_free=20387.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.307617/1.00, alarm hl:np_load_long=0.868164/1.50, alarm hl:mem_free=20387.000000M/300M, alarm hl:available=1/0 [17:46:22] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [17:47:21] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [17:48:22] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:49:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:50:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:50:55] 3(commented) [OSM-9] Periodic "Connection refused" on tirex mod_tile socket <10https://jira.toolserver.org/browse/OSM-9> (Kai Krueger) [17:51:22] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:52:22] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [17:53:32] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 461391 MB (8% inode=44%): [17:58:55] 3(commented) [OSM-9] Periodic "Connection refused" on tirex mod_tile socket <10https://jira.toolserver.org/browse/OSM-9> (Kai Krueger) [18:03:22] Is it possible to get an email notification every time something happens to a jira ticket one participates in? [18:08:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk FTOL, far1-n1-fast2 FTOL, far1-n1-fast CRIT [18:18:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75808.000000 [18:21:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75963.000000 [18:22:51] apmon: this happens for me - so yes [18:23:01] not sure how to get it that way, though [18:23:30] thanks, I'll see if I can find it in the settings somewhere [18:33:22] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.132812/1.10, alarm hl:np_load_long=0.835938/1.55, alarm hl:mem_free=21515.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.132812/1.00, alarm hl:np_load_long=0.835938/1.50, alarm hl:mem_free=21515.000000M/300M, alarm hl:available=1/0 [18:34:23] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [18:45:23] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.620117/1.10, alarm hl:np_load_long=0.963867/1.55, alarm hl:mem_free=20715.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.620117/1.00, alarm hl:np_load_long=0.963867/1.50, alarm hl:mem_free=20715.000000M/300M, alarm hl:available=1/0 [18:45:29] I just accidentally truncated ~erwin85/public_html/randomarticle.php probably cause of exceeding a quotom. Is it possible to restore that file from a TS back-up? [18:47:23] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [18:48:23] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:49:22] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:49:38] Hmm... nm. I got an automatic e-mail saying I can recover the editor buffer. [18:50:02] All seems fine now. (Besides me not making back-ups.) [18:50:02] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:51:32] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:51:39] and not cleaning up ~/ ;-) [18:53:43] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 460846 MB (8% inode=44%): [19:01:43] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:11:12] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:11:22] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=1.005859/1.00, alarm hl:np_load_long=0.762207/1.50, alarm hl:mem_free=952.000000M/300M, alarm hl:available=1/0 [19:12:22] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.163086/1.8, alarm hl:np_load_avg=1.067871/2.3, alarm hl:mem_free=206.000000M/300M, alarm hl:available=1/0 [19:12:22] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [19:13:22] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [19:18:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79411.000000 [19:22:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79623.000000 [19:22:55] 3(created) [TS-1333] Install ack; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1333> (Krinkle) [19:37:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.326172/1.10, alarm hl:np_load_long=1.218750/1.55, alarm hl:mem_free=19304.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.326172/1.00, alarm hl:np_load_long=1.218750/1.50, alarm hl:mem_free=19304.000000M/300M, alarm hl:available=1/0 [19:38:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk FTOL, far1-n1-fast2 FTOL, far1-n1-fast CRIT [19:41:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [19:45:03] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.385742/1.10, alarm hl:np_load_long=1.235351/1.55, alarm hl:mem_free=19001.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.385742/1.00, alarm hl:np_load_long=1.235351/1.50, alarm hl:mem_free=19001.000000M/300M, alarm hl:available=1/0 [19:48:00] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [19:49:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:49:02] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [19:50:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:50:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:52:02] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:54:02] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 460211 MB (8% inode=44%): [20:00:53] 3(updated) [ACCAPP-481] I am active user in http://fa.wiktionary.org and i want this account to run bot on this wiki also updating it's statics. <10https://jira.toolserver.org/browse/ACCAPP-481> (Darafsh Kaviyani) [20:02:54] 3(created) [UTRS-87] Log timestamps are sometimes incorrect; UTRS: Main Interface; Bug <10https://jira.toolserver.org/browse/UTRS-87> (Chris Howie) [20:02:56] 3(commented) [UTRS-83] Comments cut off in appeals <10https://jira.toolserver.org/browse/UTRS-83> (Chris Howie) [20:02:59] 3(commented) [ACCAPP-481] I am active user in http://fa.wiktionary.org and i want this account to run bot on this wiki also updating it's statics. <10https://jira.toolserver.org/browse/ACCAPP-481> (Darafsh Kaviyani) [20:04:57] 3(closed) [UTRS-79] Sent email content in an appeal's log discloses the appealing user's email address <10https://jira.toolserver.org/browse/UTRS-79> (Chris Howie) [20:04:58] 3(commented) [UTRS-79] Sent email content in an appeal's log discloses the appealing user's email address <10https://jira.toolserver.org/browse/UTRS-79> (Chris Howie) [20:08:00] Virtual disks on far1-n1-oe16-esams.mgmt is OK: OK 4, WARN 0, CRIT 0: far1-n1-fast3 FTOL, far1-n1-bulk FTOL, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:13:01] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.260742/1.8, alarm hl:np_load_avg=1.356445/2.3, alarm hl:mem_free=190.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=1.260742/1.9, alarm hl:np_load_long=1.145020/2.25, alarm hl:mem_free=190.000000M/200M, alarm hl:available=1/0 [20:17:01] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [20:19:01] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83031.000000 [20:22:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83226.000000 [20:26:03] Platonides * Re: [Toolserver-l] [Wikitech-l] 403: User account expired toolserver.org/~soxred93 [20:31:03] Platonides * Re: [Toolserver-l] [Wikitech-l] 403: User account expired toolserver.org/~soxred93 [20:36:06] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.097656/1.00, alarm hl:np_load_long=0.827148/1.50, alarm hl:mem_free=19717.000000M/300M, alarm hl:available=1/0 [20:48:26] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [20:49:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:50:26] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:50:34] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:52:05] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [20:52:26] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:54:05] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 459663 MB (8% inode=44%): [21:00:05] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.695801/1.8, alarm hl:np_load_avg=0.780762/2.3, alarm hl:mem_free=280.000000M/300M, alarm hl:available=1/0 [21:04:05] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:19:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86631.000000 [21:22:06] bonne nuit [21:22:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86841.000000 [21:25:06] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.035156/1.00, alarm hl:np_load_long=0.892578/1.50, alarm hl:mem_free=19876.000000M/300M, alarm hl:available=1/0 [21:26:05] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [21:30:06] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.011719/1.00, alarm hl:np_load_long=0.914062/1.50, alarm hl:mem_free=19300.000000M/300M, alarm hl:available=1/0 [21:31:04] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=1.353516/1.8, alarm hl:np_load_avg=0.912109/2.3, alarm hl:mem_free=225.000000M/300M, alarm hl:available=1/0 [21:33:04] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [21:39:03] [[w:en:User:Madman]] * [Toolserver-l] Dementia (was: 403: User account expired toolserver.org/~soxred93) [21:49:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:49:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [21:51:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:51:26] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:53:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:54:05] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 459115 MB (8% inode=44%): [22:00:38] @replag [22:00:39] Jeff|mobile: s1-rr-a: 1d 45m 34s [+1.00 s/s]; s1-user: 1d 45m 34s [+1.00 s/s]; s3-rr-a: 1m 5s [+0.00 s/s]; s3-user: 1m 5s [+0.00 s/s] [22:01:34] s1 is lagging almost 25 hours? [22:04:18] Jeff|mobile: s1 is not updating, apparently [22:04:43] http://status.toolserver.org/ [22:04:50] sql-s1: INFO MNT-1225 alter table enwiki by wmf [22:06:01] Thanks [22:10:56] https://jira.toolserver.org/browse/MNT-1225 [22:19:09] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90236.000000 [22:20:12] What? [22:20:46] valhallasw: Decentralizing status info is bad. :-) [22:21:28] Joan: having to click on a link to get info is bad too [22:21:39] if you have problems with manually updating it, write a bot to do it [22:22:00] Why is cliking a link bad? [22:22:02] clicking [22:22:49] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90462.000000 [22:27:07] because it takes more effort [22:45:16] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.214844/1.10, alarm hl:np_load_long=0.963867/1.55, alarm hl:mem_free=20529.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.214844/1.00, alarm hl:np_load_long=0.963867/1.50, alarm hl:mem_free=20529.000000M/300M, alarm hl:available=1/0 [22:46:16] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [22:46:46] could I ask if it is possible for a query to check for identical filenames on commons? [22:47:39] ToAruShiroiNeko: You mean to en.wp? [22:47:55] no commons [22:48:06] like image.jpg and image.JPG [22:48:14] identical text with case difference [22:48:26] could also be image.jPg or even iMage.jpg [22:48:45] I want to list all conflicts and link to it on village pump [22:48:47] Ah. Hmmm... [22:49:06] its unhelpful when you have indentical filenames with only case difference [22:49:16] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:49:27] thread is at http://commons.wikimedia.org/wiki/Commons:Village_pump#Rename_identically_named_different_files [22:49:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [22:49:52] some people commented a more broader search but thats perhaps a later consideration [22:50:00] select * from page where page_namespace=0 and page_title="albert_speer" collate latin1_general_ci limit 1; [22:50:06] something like that, but by joining page to page [22:50:28] There are other cases like File:Ｐｉｃｔｕｒｅ.jpg and File:Picture.jpg but thats a much later issue [22:50:39] ToAruShiroiNeko: how are those the same? [22:50:53] it is different encoding of same text [22:50:57] if it's a duplicate image (i.e. the same data) you can compare the sha1 hash [22:51:06] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:51:11] I meant filenames valhallasw [22:51:30] anyways my main worry is non-unicode filenames that read identical with only case difference [22:51:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:52:30] you're not making much sense [22:52:40] what the fuck are 'non-unicode filenames' [22:52:46] valhallasw: It sounds like a regex thing... [22:52:48] every character is 'unicode' [22:52:55] matthewrbowker: no, it's just using the right collation [22:53:46] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:54:17] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 458946 MB (8% inode=44%): [22:54:20] Hmmm.. OK :P [22:54:57] the full-width P is more problematic, as it doesn't seem to be handled by general collations [22:58:16] Sun Grid Engine execd on ortelius is WARNING: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.055664/1.00, alarm hl:np_load_long=0.941406/1.50, alarm hl:mem_free=20856.000000M/300M, alarm hl:available=1/0 [22:59:07] select p1.page_title, p2.page_title from page as p1 left join page as p2 on p1.page_namespace=p2.page_namespace and p1.page_title=p2.page_title collate latin1_general_ci where p1.page_namespace=6 and p1.page_title<>p2.page_title limit 1; [22:59:11] something like that [22:59:22] but it's of course slow as hell :P [23:01:10] Oh. [23:01:17] I already have a report for that, ToAruShiroiNeko. [23:01:22] It's at Commons:Database reports. [23:02:00] you mean http://commons.wikimedia.org/wiki/Commons:Database_reports/Largely_duplicative_file_names ? [23:02:01] Oh, it's also broken: https://commons.wikimedia.org/wiki/Commons:Database_reports/Largely_duplicative_file_names [23:02:05] yes. [23:02:12] When it's working, it does what you want. [23:02:19] how is 'Largely duplicative' defined? [23:02:22] I think it can't get all the data from Commons in one query any more. [23:03:16] valhallasw: https://en.wikipedia.org/wiki/wp:Database_reports/Largely_duplicative_file_names [23:03:23] Wikipedia is slow as fuck for me right now. [23:03:28] Configuration is at ... [23:03:31] https://en.wikipedia.org/wiki/wp:Database_reports/Largely_duplicative_file_names/Configuration [23:04:53] Joan: creating the index after inserting data is supposed to be faster [23:05:35] your code is slightly more correct than just the collation in that is also works for the 'japanese' P/p [23:05:47] but it doesn't detect P=Ｐ either [23:05:58] Yeah. [23:06:02] You'd need noramlization for that. [23:06:07] There are modules for that kind of thing. [23:06:14] unicode normalization doesn't do that [23:06:19] You can quickly get a lot of false positives, though. [23:06:29] I'm not sure if "normalization" is the word. [23:06:30] normalization in principle only changes the representation [23:06:44] You can get things that will convert é to e. [23:06:45] collation is about comparison, but in my quick check they are still different [23:06:47] And things like that. [23:07:41] E0050 ; [.0000.0000.0000.0000] # [E0050] TAG LATIN CAPITAL LETTER P [23:07:52] versus... [23:08:07] FF30 ; [.177B.0020.0009.FF30] # FULLWIDTH LATIN CAPITAL LETTER P [23:08:18] but now I have to check what the meaning of those number is again :P [23:09:10] FF50 ; [.177B.0020.0003.FF50] # FULLWIDTH LATIN SMALL LETTER P [23:09:33] from left to right you have each time a more specific comparison [23:10:38] see http://unicode.org/reports/tr10/#Multi_Level_Comparison [23:12:10] http://stackoverflow.com/questions/517923/what-is-the-best-way-to-remove-accents-in-a-python-unicode-string would be interesting. [23:12:25] I imagine Commoners would get angry about considering "Foö.jpg" the same as "Foo.jpg", though. [23:13:00] again, read up on collations [23:13:16] Sun Grid Engine execd on willow is WARNING: medium-sol@willow exceedes load threshold: alarm hl:np_load_short=0.960938/1.8, alarm hl:np_load_avg=1.011719/2.3, alarm hl:mem_free=159.000000M/300M, alarm hl:available=1/0: longrun-sol@willow exceedes load threshold: alarm hl:np_load_short=0.960938/1.9, alarm hl:np_load_long=0.896973/2.25, alarm hl:mem_free=159.000000M/200M, alarm hl:available=1/0 [23:14:51] ToAruShiroiNeko should read up. ;-) [23:15:16] Sun Grid Engine execd on willow is OK: medium-sol@willow OK: longrun-sol@willow OK [23:15:22] but you can use NFKC for this [23:15:56] In general, I'd recommend not relying on MySQL for any of this, though. [23:16:55] it would be fine if the columns were utf-8 [23:17:00] and not the crap they are now :p [23:17:47] ok, so apparently the thing to do is first NFKC normalization and then comparing stuff with the right collation [23:18:17] NFKC makes the Ｐ become a P, the collation (depending on setting) makes P=p, é=e, etc [23:19:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93842.000000 [23:19:43] Collation affects case? [23:20:12] no [23:20:16] comparison and sorting [23:22:57] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94067.000000 [23:42:03] Platonides * Re: [Toolserver-l] Dementia (was: 403: User account expired toolserver.org/~soxred93) [23:44:17] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.106445/1.10, alarm hl:np_load_long=0.866211/1.55, alarm hl:mem_free=20114.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.106445/1.00, alarm hl:np_load_long=0.866211/1.50, alarm hl:mem_free=20114.000000M/300M, alarm hl:available=1/0 [23:45:17] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [23:49:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:49:56] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default [23:51:16] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:51:45] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:53:57] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:54:17] /aux0 on hemlock is WARNING: DISK WARNING - free space: /aux0 458430 MB (8% inode=44%): [23:54:17] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.237305/1.10, alarm hl:np_load_long=0.874023/1.55, alarm hl:mem_free=20167.000000M/300M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.237305/1.00, alarm hl:np_load_long=0.874023/1.50, alarm hl:mem_free=20167.000000M/300M, alarm hl:available=1/0