[00:03:04] Load avg. on willow is WARNING: WARNING - load average: 15.62, 16.75, 14.85 [00:12:15] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.260742/1.10, alarm hl:np_load_long=1.579101/1.55, alarm hl:mem_free=19002.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.260742/1.00, alarm hl:np_load_long=1.579101/1.50, alarm hl:mem_free=19002.000000M/300M, alarm hl:available=1/0 [00:14:15] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [00:18:14] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.024414/1.00, alarm hl:np_load_long=1.414062/1.50, alarm hl:mem_free=19184.000000M/300M, alarm hl:available=1/0 [00:19:15] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [00:20:04] Load avg. on willow is OK: OK - load average: 12.70, 14.40, 14.98 [00:23:05] Load avg. on willow is WARNING: WARNING - load average: 13.08, 14.56, 15.01 [00:28:15] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:29:05] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:29:14] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:30:46] nacht ts [00:32:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:44:48] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:45:53] 3(resolved) [UTRS-62] Bot Text Colors <10https://jira.toolserver.org/browse/UTRS-62> (Andrew Pearson) [00:47:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 35407 MB (3% inode=99%): [00:54:18] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [01:12:39] Load avg. on willow is WARNING: WARNING - load average: 14.52, 15.83, 14.76 [01:14:38] Load avg. on willow is OK: OK - load average: 13.30, 14.98, 14.57 [01:28:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:29:40] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:29:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:31:40] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37816 MB (9% inode=99%): [01:32:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:34:39] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 61463 MB (15% inode=99%): [01:37:33] [[User:Dab/Debian-Packages]] ! 10https://wiki.toolserver.org/w/index.php?diff=6767&oldid=6766&rcid=8934 * 163.1.163.253 * (+12) (/* General Tools & Development */ librsvg2-2) [01:40:38] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39577 MB (9% inode=99%): [02:02:51] Load avg. on willow is WARNING: WARNING - load average: 15.50, 15.84, 14.66 [02:04:51] Load avg. on willow is OK: OK - load average: 13.48, 14.97, 14.48 [02:06:00] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.132812/1.10, alarm hl:np_load_long=0.953125/1.55, alarm hl:mem_free=19063.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.132812/1.00, alarm hl:np_load_long=0.953125/1.50, alarm hl:mem_free=19063.000000M/300M, alarm hl:available=1/0 [02:06:36] @replag [02:06:37] addshore: s3-rr-a: 1m 54s [+0.00 s/s]; s3-user: 1m 54s [+0.00 s/s]; s5-rr-a: 15s [+0.00 s/s]; s5-user: 15s [+0.00 s/s] [02:08:10] [[User:Dab/Debian-Packages]] ! 10https://wiki.toolserver.org/w/index.php?diff=6768&oldid=6767&rcid=8935 * 163.1.163.253 * (-12) (changed mind) [02:12:48] Load avg. on willow is WARNING: WARNING - load average: 14.86, 15.17, 14.74 [02:17:38] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 36407 MB (8% inode=99%): [02:21:37] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 22583 MB (5% inode=99%): [02:23:37] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 49100 MB (12% inode=99%): [02:26:59] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [02:28:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:30:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:30:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:33:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:38:10] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.509766/1.10, alarm hl:np_load_long=1.357422/1.55, alarm hl:mem_free=19342.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.509766/1.00, alarm hl:np_load_long=1.357422/1.50, alarm hl:mem_free=19342.000000M/300M, alarm hl:available=1/0 [02:47:49] Load avg. on willow is WARNING: WARNING - load average: 13.80, 15.02, 15.33 [02:49:49] Load avg. on willow is OK: OK - load average: 11.53, 13.70, 14.79 [02:54:09] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [03:02:50] Load avg. on willow is WARNING: WARNING - load average: 16.57, 16.23, 15.12 [03:03:08] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.257812/1.10, alarm hl:np_load_long=1.264649/1.55, alarm hl:mem_free=19362.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.257812/1.00, alarm hl:np_load_long=1.264649/1.50, alarm hl:mem_free=19362.000000M/300M, alarm hl:available=1/0 [03:04:50] Load avg. on willow is OK: OK - load average: 11.79, 14.73, 14.70 [03:07:08] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [03:10:15] Does anyone know if I can get my crontab file off of nightshade? [03:10:28] I'm trying to move all of my cron tasks to the cronie system. [03:16:28] nightshade was wiped. [03:16:41] So if you don't have a backup, you're SOL, I think. [03:16:57] You should save your crontab to a text file and sync it with "cronie ~/path/to/crontab.txt". [03:17:12] So that it'll get backed up with other files and reside on /home. [03:29:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:30:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:30:58] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:33:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:44:52] 3(created) [ACCAPP-469] Help development with CVN (and commit access to p_cvn), possibly run query services; Account Approval; New Account <10https://jira.toolserver.org/browse/ACCAPP-469> (Daniel Salciccioli) [03:52:52] :( [03:55:29] Load avg. on ortelius is WARNING: WARNING - load average: 16.30, 12.97, 9.43 [03:56:28] Load avg. on ortelius is OK: OK - load average: 13.10, 12.62, 9.54 [03:57:42] Never trust a file you can't touch! [03:58:08] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.832031/1.10, alarm hl:np_load_long=2.296875/1.55, alarm hl:mem_free=18995.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.832031/1.00, alarm hl:np_load_long=2.296875/1.50, alarm hl:mem_free=18995.000000M/300M, alarm hl:available=1/0 [04:01:38] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37836 MB (9% inode=99%): [04:01:52] 3(work started) [MATTHEWRBOWKER-7] Lists threads running on command <10https://jira.toolserver.org/browse/MATTHEWRBOWKER-7> (Matthew Bowker) [04:02:50] Load avg. on willow is WARNING: WARNING - load average: 18.56, 17.46, 15.55 [04:03:37] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 57981 MB (14% inode=99%): [04:10:37] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 37557 MB (9% inode=99%): [04:19:51] 3(resolved) [MATTHEWRBOWKER-7] Lists threads running on command <10https://jira.toolserver.org/browse/MATTHEWRBOWKER-7> (Matthew Bowker) [04:21:52] 3(created) [MATTHEWRBOWKER-8] On-IRC help system for WikiWelcomer; Matthewrbowker's Tools: WikiWelcomer; Critical New Feature <10https://jira.toolserver.org/browse/MATTHEWRBOWKER-8> (Matthew Bowker) [04:24:48] Load avg. on willow is OK: OK - load average: 12.08, 13.75, 14.96 [04:29:09] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:30:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:30:58] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:33:08] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [04:34:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:36:11] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.133789/1.10, alarm hl:np_load_long=1.375976/1.55, alarm hl:mem_free=17773.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.133789/1.00, alarm hl:np_load_long=1.375976/1.50, alarm hl:mem_free=17773.000000M/300M, alarm hl:available=1/0 [04:42:11] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [05:27:25] [[Special:Log/newusers]] create 10 * Tarunno * (New user account) [05:27:51] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39205 MB (9% inode=99%): [05:28:52] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 65247 MB (16% inode=99%): [05:29:21] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.468750/1.10, alarm hl:np_load_long=1.438476/1.55, alarm hl:mem_free=18276.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.468750/1.00, alarm hl:np_load_long=1.438476/1.50, alarm hl:mem_free=18276.000000M/300M, alarm hl:available=1/0 [05:29:21] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:30:21] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:31:01] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:31:52] Load avg. on willow is WARNING: WARNING - load average: 18.62, 16.62, 14.76 [05:34:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:37:00] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 35672 MB (8% inode=99%): [05:37:01] Load avg. on willow is OK: OK - load average: 13.04, 14.61, 14.45 [05:45:18] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [05:51:18] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.010742/1.00, alarm hl:np_load_long=1.309570/1.50, alarm hl:mem_free=18763.000000M/300M, alarm hl:available=1/0 [05:52:18] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [05:53:01] Load avg. on willow is WARNING: WARNING - load average: 15.54, 15.10, 14.40 [05:58:00] Load avg. on willow is OK: OK - load average: 13.68, 14.89, 14.59 [06:02:59] Load avg. on willow is WARNING: WARNING - load average: 13.21, 15.17, 14.88 [06:17:40] [[Special:Log/newusers]] create 10 * Forstbirdo * (New user account) [06:30:18] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.035156/1.00, alarm hl:np_load_long=0.976562/1.50, alarm hl:mem_free=18716.000000M/300M, alarm hl:available=1/0 [06:30:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:31:08] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:31:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:35:20] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [06:35:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:48:59] Load avg. on willow is WARNING: WARNING - load average: 12.82, 15.78, 16.30 [07:03:43] Is there any way to change the max length of a job submitted to SGE after submission? [07:04:03] To avoid a job longer than expected to be killed and trashed. [07:06:58] Load avg. on willow is CRITICAL: CRITICAL - load average: 31.54, 23.57, 19.17 [07:07:58] Load avg. on willow is WARNING: WARNING - load average: 23.52, 22.71, 19.14 [07:09:58] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.74, 23.84, 20.01 [07:16:18] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=3.207031/1.10, alarm hl:np_load_long=1.166992/1.55, alarm hl:mem_free=18800.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=3.207031/1.00, alarm hl:np_load_long=1.166992/1.50, alarm hl:mem_free=18800.000000M/300M, alarm hl:available=1/0 [07:20:18] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [07:20:22] oh, Merlissimo, thanks for expanding the docs! [07:23:17] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.001953/1.00, alarm hl:np_load_long=1.139649/1.50, alarm hl:mem_free=18851.000000M/300M, alarm hl:available=1/0 [07:29:58] Load avg. on willow is WARNING: WARNING - load average: 17.91, 16.31, 17.69 [07:30:28] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:31:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:33:34] Nemo_bis: changing the job runtime after submition is not allowed. otherwise you could have your job running at a queue with very high cpu priority forever [07:36:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:39:14] Merlissimo, yes, makes sense [07:39:18] thank you [07:39:49] although perhaps one could be allowed to change his mind once for a job, in an ideal world [07:42:40] [[Job scheduling]] ! 10https://wiki.toolserver.org/w/index.php?diff=6769&oldid=6755&rcid=8938 * Nemobis * (+257) (/* obligatory resources */ runtime limit, per Merlissimo) [07:44:26] It's good that your working on docs, because AFAICS SGE is used way less than it should. [07:49:42] i like sge because people can submit a jobs and don't have to care about the system where the job is executed. unfortunately the ts admins to not know sge and solaris so good. i am a bit afraid on using debian on a multi user system with no resource control [07:52:15] Merlissimo, a simple thing that would be useful is a plain list of the SGE commands. There is plenty of them and one doesn't even know where to look for man pages. [07:55:52] 3(updated) [MNT-1210] Willow run havoc <10https://jira.toolserver.org/browse/MNT-1210> (Marlen Caemmerer) [07:55:57] 3(resolved) [TS-1321] No updates for plwiki_p since 2012-02-28 <10https://jira.toolserver.org/browse/TS-1321> (Marlen Caemmerer) [07:57:52] 3(resolved) [TS-1287] Database S5 extremely slow <10https://jira.toolserver.org/browse/TS-1287> (Marlen Caemmerer) [07:57:54] 3(resolved) [MNT-1192] s3 and s7 slaves were stopped <10https://jira.toolserver.org/browse/MNT-1192> (Marlen Caemmerer) [08:13:17] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.622070/1.10, alarm hl:np_load_long=1.075195/1.55, alarm hl:mem_free=18863.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.622070/1.00, alarm hl:np_load_long=1.075195/1.50, alarm hl:mem_free=18863.000000M/300M, alarm hl:available=1/0 [08:25:19] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [08:30:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:30:53] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6770&oldid=6768&rcid=8939 * Valhallasw * (+197) (/* Python */ +python2.7) [08:30:58] Load avg. on willow is WARNING: WARNING - load average: 25.86, 21.03, 18.93 [08:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:31:18] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:36:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:57:56] 3(created) [OSM-9] Periodic "Connection refused" on tirex mod_tile socket; OSM; Minor Bug <10https://jira.toolserver.org/browse/OSM-9> (Kai Krueger) [09:01:09] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.44, 19.03, 16.94 [09:02:09] Load avg. on willow is WARNING: WARNING - load average: 24.30, 19.46, 17.24 [09:17:28] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.337891/1.10, alarm hl:np_load_long=1.060547/1.55, alarm hl:mem_free=18422.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.337891/1.00, alarm hl:np_load_long=1.060547/1.50, alarm hl:mem_free=18422.000000M/300M, alarm hl:available=1/0 [09:19:28] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [09:25:29] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.083985/1.10, alarm hl:np_load_long=1.123047/1.55, alarm hl:mem_free=18286.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.083985/1.00, alarm hl:np_load_long=1.123047/1.50, alarm hl:mem_free=18286.000000M/300M, alarm hl:available=1/0 [09:30:38] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:31:09] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:31:28] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:36:18] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:36:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:37:09] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:39:09] Load avg. on willow is OK: OK - load average: 11.71, 13.15, 14.97 [09:43:09] Load avg. on willow is WARNING: WARNING - load average: 13.02, 14.17, 15.07 [09:45:09] Load avg. on willow is OK: OK - load average: 12.49, 13.79, 14.82 [09:58:11] [[Category:Edit counters]] ! 10https://wiki.toolserver.org/w/index.php?diff=6771&oldid=6698&rcid=8940 * 121.241.235.68 * (+56) () [10:03:28] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=4.858398/1.10, alarm hl:np_load_long=1.447266/1.55, alarm hl:mem_free=17673.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=4.858398/1.00, alarm hl:np_load_long=1.447266/1.50, alarm hl:mem_free=17673.000000M/300M, alarm hl:available=1/0 [10:07:28] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [10:27:28] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.063476/1.00, alarm hl:np_load_long=1.060547/1.50, alarm hl:mem_free=18486.000000M/300M, alarm hl:available=1/0 [10:30:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:31:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:31:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:37:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:46:03] @replag all [10:46:04] Sebaso_WMDE: s1-rr-a: 0s [-0.00 s/s]; s1-rr-a-c: 36s [+0.03 s/s]; s1-user: 0s [-0.00 s/s]; s2-user: 12s [-]; s2-user-c: 0s [-]; s3-rr-a: 1m 24s [+0.07 s/s]; s3-user: 1m 25s [+0.07 s/s]; s4-rr-a: 1s [+0.00 s/s] [10:46:05] Sebaso_WMDE: s4-user: 1s [+0.00 s/s]; s5-rr-a: 1s [-0.00 s/s]; s5-user: 1s [-0.00 s/s]; s5-user-c: 1s [+0.00 s/s]; s6-rr-a: 6s [+0.00 s/s]; s6-user: 6s [+0.00 s/s]; s7-rr-a: 9s [+0.01 s/s]; s7-user: 9s [+0.01 s/s] [11:06:08] Load avg. on willow is WARNING: WARNING - load average: 18.26, 16.73, 15.54 [11:18:08] Load avg. on willow is OK: OK - load average: 12.00, 14.12, 14.90 [11:30:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:31:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:31:38] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:37:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:04:09] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38773 MB (9% inode=99%): [12:08:08] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 53996 MB (13% inode=99%): [12:14:07] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 38783 MB (9% inode=99%): [12:30:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:31:19] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:31:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:37:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:43:53] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.031250/1.00, alarm hl:np_load_long=0.901367/1.50, alarm hl:mem_free=17994.000000M/300M, alarm hl:available=1/0 [12:44:12] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39646 MB (9% inode=99%): [12:44:54] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [12:47:12] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 63489 MB (15% inode=99%): [12:48:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 35364 MB (3% inode=99%): [12:50:22] Sun Grid Engine execd on wolfsbane is WARNING: short@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.234863/1.10, alarm hl:np_load_long=0.208008/1.55, alarm hl:mem_free=128.000000M/300M, alarm hl:available=1/0: all.q@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.234863/1.00, alarm hl:np_load_long=0.208008/1.50, alarm hl:mem_free=128.000000M/300M, alarm hl:available=1/0 [12:57:12] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 41198 MB (10% inode=99%): [13:06:23] Sun Grid Engine execd on wolfsbane is OK: short@wolfsbane OK: all.q@wolfsbane OK [13:08:54] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.667969/1.10, alarm hl:np_load_long=0.921875/1.55, alarm hl:mem_free=17946.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.667969/1.00, alarm hl:np_load_long=0.921875/1.50, alarm hl:mem_free=17946.000000M/300M, alarm hl:available=1/0 [13:10:53] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [13:30:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:31:22] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:31:54] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:37:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:13:52] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.345703/1.10, alarm hl:np_load_long=0.844727/1.55, alarm hl:mem_free=17974.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.345703/1.00, alarm hl:np_load_long=0.844727/1.50, alarm hl:mem_free=17974.000000M/300M, alarm hl:available=1/0 [14:14:53] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [14:19:52] [[Toolserver:Autoconfirmed users]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=6772&rcid=8941 * Eseki * (+33) (Created page with "--~~~~") [14:22:46] [[Toolserver:Ts-users]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=6773&rcid=8942 * Eseki * (+33) (Created page with "--~~~~") [14:26:16] hello all [14:26:34] [[User:Eseki]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=6774&rcid=8943 * Eseki * (+33) (Created page with "--~~~~") [14:26:59] Morning DaB [14:28:44] [[System administrators]] !M 10https://wiki.toolserver.org/w/index.php?diff=6775&oldid=6402&rcid=8944 * Eseki * (+34) (/* admins */ ) [14:31:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:31:32] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:32:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:32:54] @replag [14:32:54] DaBPunkt: s2-user: 11s [-0.00 s/s]; s3-rr-a: 24s [-0.00 s/s]; s3-user: 24s [-0.00 s/s] [14:38:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:41:22] [[System administrators]] ! 10https://wiki.toolserver.org/w/index.php?diff=6776&oldid=6775&rcid=8945 * Hydra * (-34) (Undo revision 6775 by [[Special:Contributions/Eseki|Eseki]] ([[User talk:Eseki|talk]])) [14:41:59] [[Toolserver:Ts-users]] ! 10https://wiki.toolserver.org/w/index.php?diff=6777&oldid=6773&rcid=8946 * Hydra * (+32) (Delete - Little or no meaning) [14:42:08] [[Toolserver:Autoconfirmed users]] ! 10https://wiki.toolserver.org/w/index.php?diff=6778&oldid=6772&rcid=8947 * Hydra * (+32) (Delete - Little or no meaning) [15:02:23] Load avg. on willow is WARNING: WARNING - load average: 16.34, 15.53, 14.24 [15:03:45] [[Special:Log/delete]] delete 10 * Dab * (deleted "[[02Toolserver:Ts-users10]]": non-use) [15:04:05] [[Special:Log/delete]] delete 10 * Dab * (deleted "[[02Toolserver:Autoconfirmed users10]]": nonsense) [15:07:22] Load avg. on willow is OK: OK - load average: 14.62, 14.95, 14.34 [15:12:22] Load avg. on willow is WARNING: WARNING - load average: 14.87, 15.77, 14.89 [15:31:42] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:32:03] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:32:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:39:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:39:48] I noticed something strange with the enwiki databse. Is this more likely to be a mediawiki bug or toolserver data corruption? There are some articles that have category membership records in the toolserver db that are not visible on enwiki. If I blank the page on enwiki, the other categories do disappear, and the toolserver db does show a page_len of 0, but the bad categories are still shown (even though the page is empty). When I undo the page blan [15:48:56] carl-m: did you check if the page shows up in the "bad" categories on enwiki? [15:49:03] DaB. * Re: [Toolserver-l] Debian (Linux) is coming back [15:49:28] the thing is - when you edit the article, all categorylinks should be refreshed, and that update should propagate to the toolserver [15:49:35] can't really see how it wouldn't happen. [15:50:45] Daniel_WMDE_: it did not show up even when I purged the page and then loaded it as a logged-out user [15:51:28] and I have blanked the page and unblanked it twice, so like you said I expected the links would be refreshed. Also by blanking and unblanking I can be sure that the changes have replicated because I can see that the other categories disappear and reappear on toolserver [15:54:39] this is quite strange [15:54:46] please file a ticket on jira [15:55:02] i'm not sure if it'S a toolserver bug, but i think it should definitly be documented somewhere [15:55:05] and provider an example [15:55:16] yes, please provide the concrete query [15:55:48] i found category links with cl_from = 0 recently, on dewiki i think [15:56:03] but i believe these are actual spurious entries in the live db [15:56:03] [[User:Dab/Debian-Packages]] ! 10https://wiki.toolserver.org/w/index.php?diff=6779&oldid=6770&rcid=8950 * 192.12.184.7 * (+73) (/* General Tools & Development */ ) [15:58:42] [[User:Dab/Debian-Packages]] M 10https://wiki.toolserver.org/w/index.php?diff=6780&oldid=6779&rcid=8951 * Dab * (-5) () [16:02:24] @replag [16:02:24] DaBPunkt: s1-rr-a-c: 13s [-0.00 s/s]; s1-user: 17s [+0.00 s/s]; s2-user-c: 26s [+0.00 s/s]; s3-rr-a: 11s [-0.00 s/s]; s3-user: 11s [-0.00 s/s]; s5-user-c: 26s [+0.00 s/s] [16:05:09] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6781&oldid=6780&rcid=8952 * Dab * (+28) (/* General Tools & Development */ already in base) [16:06:01] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6782&oldid=6781&rcid=8953 * Dab * (+26) () [16:07:03] Daniel_WMDE_: can you query the live db? The query is select page_namespace, page_title, page_len, cl_to from page join categorylinks on page_id = cl_from where page_namespace = 1 and page_title like 'Henderson_Home%' [16:07:31] you could make that = 'Henderson_Home_News' to get rid of the like [16:08:18] there ought to be 11 categories. Anyway, I will file it in JIRA if there is some chance it could be a toolserver bug [16:08:41] carl-m: no, i don't have access [16:08:55] ask in #wikimedia-tech [16:09:23] Since the cats don't show up on the live site, that makes me think it is a toolserver issue. I remember there was some anomaly in the past few months which they said could have lead to corruption [16:10:28] [[User:Dab/Debian-Packages]] ! 10https://wiki.toolserver.org/w/index.php?diff=6783&oldid=6782&rcid=8954 * 212.202.143.197 * (+6) (/* General Tools & Development */ +curl) [16:11:36] carl-m: the query reteurns the same on the TS and on the live-servers [16:12:02] DaBPunkt: thanks - does it give 14 results on both? [16:12:09] no, 11 [16:12:12] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.072266/1.00, alarm hl:np_load_long=0.870117/1.50, alarm hl:mem_free=18497.000000M/300M, alarm hl:available=1/0 [16:13:10] I have a sql window open on toolserver, and I get 14 results when I run it right now [16:13:11] oh, thyme shows 14 results [16:13:12] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [16:14:01] mm, I think that shows that thymes copy is corrupt :( [16:16:00] yep [16:16:36] I remember the announcement that there was a replication error recently, could that have caused this? [16:16:48] yes [16:17:42] that makes sense. do you want me to file something in jira or is it already a known issue? [16:18:36] make a bugreport please [16:26:12] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6784&oldid=6783&rcid=8955 * Dab * (+0) (that's c) [16:27:05] DaBPunkt: a few people wrote to offer solaris support; i don't know solaris but if you want more debian people i'm happy to poke at some stuff. [maybe it's already published, idk?] would be nice if the puppet repo were public so that general public could help with puppet changes. e.g. porting lists on wiki pages over to puppet and implementing stuff from JIRA tickets [16:27:59] 3(created) [TS-1326] Database corruption in enwiki_p on thyme; Toolserver; Minor Bug <10https://jira.toolserver.org/browse/TS-1326> (CBM) [16:28:27] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6785&oldid=6784&rcid=8956 * Dab * (+28) (/* General Tools & Development */ ) [16:29:31] jeremyb: yes, the problem is that our puppet containse some passwords at the moment and I haven't looked at all files yet. So publishing everything at the moment is not possible (I very like the grrrit-system the wmf-uses) [16:29:50] the wmf uses [16:31:06] DaBPunkt: sure, you could probably even use their gerrit. but gerrit and separating out passwords to their own repo are really 2 separate tasks [16:31:42] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:31:54] jeremyb: yes, of course [16:31:56] DaBPunkt: anyway, feel free to poke if you want a hand with something. also it seems maybe you have a TZ bias for europe in the roots and i'm in US [16:32:12] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:32:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:32:29] DaBPunkt: (otoh i don't actually have a TS acct and i'm sure there's some policies i don't know) [16:32:35] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6786&oldid=6785&rcid=8957 * Dab * (+11) (/* General Tools & Development */ split) [16:33:13] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.266602/1.10, alarm hl:np_load_long=1.057617/1.55, alarm hl:mem_free=18160.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.266602/1.00, alarm hl:np_load_long=1.057617/1.50, alarm hl:mem_free=18160.000000M/300M, alarm hl:available=1/0 [16:33:45] jeremyb: sure, I will ask when I fibd a problem ) [16:33:47] :) [16:34:02] :) [16:34:32] Anyone have an efficient SQL query to find log entries for a given page? log_namespace and log_title doesn't seem to be the trick, and log_page isn't in the DB view on TS. [16:36:37] Nettrom: Request the adding of log_page in a bug-report [16:37:01] DaBPunkt: I'll do that, thanks much! :) [16:37:01] (and normaly log_namespace and log_title DOES the trick) [16:37:12] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [16:37:18] Nettrom: there is a view called 'logging_ts_alternative' that has log_page [16:37:46] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6787&oldid=6786&rcid=8958 * Dab * (+14) (/* General Tools */ in base) [16:37:54] carl-m: ooh, sweet, I'll see if that does what I need, thanks! [16:38:07] DaBPunkt: maybe a few of you want are interested in http://puppetlabs.com/community/puppet-camp/ ? anyway just FYI (coming up soon near you. or at least near your servers!) [16:38:40] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6788&oldid=6787&rcid=8959 * Dab * (-11) (/* General Tools */ not avaiable) [16:39:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:39:44] Nettrom: also see http://www.mail-archive.com/toolserver-l@lists.wikimedia.org/msg03516.html [16:41:56] Nettrom: another problem, independent of that, is that when a page is deleted and undeleted it gets a new page id, so searching by page id will not always get you all the old log actions for a particular title [16:42:53] carl-m: that was only true until mediawiki 1.11 [16:43:59] carl-m: thank for the help! [16:44:19] I get terribly slow queries for log_namespace and log_title in logging, the DB server doesn't seem to want to use the index for it [16:44:35] while log_page in logging_ts_alternative is immediate [16:44:52] Nettrom: how fast is log_title and log_namespace in the alternate table? [16:45:34] DaBPunkt: would anyone have gone back and changed the page ids for pages that were deleted before 1.11? [16:45:42] no [16:45:42] carl-m: interesting, they're instant [16:46:09] Nettrom: the alternate one is set up with a faster view than the normal one, but the cost is that some rows are hidden in the alternate one [16:46:32] carl-m: ah, that explains why the regular one isn't as quick as I expected [16:54:22] Load avg. on willow is WARNING: WARNING - load average: 17.81, 14.66, 13.46 [16:58:36] [[User:Dab/Debian-Packages]] ! 10https://wiki.toolserver.org/w/index.php?diff=6789&oldid=6788&rcid=8960 * 192.12.184.7 * (+16) (/* Python */ for the gps_exif bot on commons) [17:00:36] [[User:Dab/Debian-Packages]] ! 10https://wiki.toolserver.org/w/index.php?diff=6790&oldid=6789&rcid=8961 * 192.12.184.7 * (+39) (/* General Development */ for wikiminiatlas backend and zoomviewer) [17:02:57] [[User:Dab/Debian-Packages]] ! 10https://wiki.toolserver.org/w/index.php?diff=6791&oldid=6790&rcid=8962 * 192.12.184.7 * (+15) (/* General Development */ wma backend, rescaling of satellite images) [17:03:12] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.019531/1.00, alarm hl:np_load_long=0.953125/1.50, alarm hl:mem_free=18421.000000M/300M, alarm hl:available=1/0 [17:04:02] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6792&oldid=6791&rcid=8963 * Dab * (+84) (/* General Development */ ) [17:04:32] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6793&oldid=6792&rcid=8964 * Dab * (-66) (/* General Development */ I guess that's C(++) too) [17:04:42] [[User:Dab/Debian-Packages]] 10https://wiki.toolserver.org/w/index.php?diff=6794&oldid=6793&rcid=8965 * Dab * (+66) (/* C(++/#)/Mono */ ) [17:06:23] Load avg. on willow is OK: OK - load average: 12.77, 14.59, 14.95 [17:08:13] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [17:31:42] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:32:12] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:32:13] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:39:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:12:14] Load avg. on willow is WARNING: WARNING - load average: 18.37, 16.48, 15.05 [22:23:14] Load avg. on willow is OK: OK - load average: 12.91, 14.51, 14.86 [22:32:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:33:14] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:33:53] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:39:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:48:23] Load avg. on ortelius is WARNING: WARNING - load average: 19.87, 14.66, 8.58 [22:49:22] Load avg. on ortelius is OK: OK - load average: 13.82, 13.74, 8.63 [22:53:43] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=0.885742/1.10, alarm hl:np_load_long=1.883789/1.55, alarm hl:mem_free=18463.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=0.885742/1.00, alarm hl:np_load_long=1.883789/1.50, alarm hl:mem_free=18463.000000M/300M, alarm hl:available=1/0 [23:00:44] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [23:02:24] Load avg. on willow is WARNING: WARNING - load average: 16.53, 15.37, 14.29 [23:04:25] Load avg. on willow is OK: OK - load average: 13.64, 14.68, 14.16 [23:07:45] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=2.493164/1.10, alarm hl:np_load_long=1.428711/1.55, alarm hl:mem_free=18436.000000M/300M, alarm hl:available=1/0: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=2.493164/1.00, alarm hl:np_load_long=1.428711/1.50, alarm hl:mem_free=18436.000000M/300M, alarm hl:available=1/0 [23:11:45] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [23:12:25] Load avg. on willow is WARNING: WARNING - load average: 16.29, 15.83, 14.86 [23:33:05] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:33:25] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:34:06] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:40:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:51:57] nacht ts