[00:02:59] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.179688/1.00, alarm hl:mem_free=13232.000000M/100M [00:06:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:08:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:08:58] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [00:13:29] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.947266/2.00, alarm hl:mem_free=224.000000M/250M [00:15:49] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199340 MB (3% inode=26%): [00:16:07] /sql on adenia is WARNING: DISK WARNING - free space: /sql 68659 MB (10% inode=99%): [00:16:28] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:20:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 65038 MB (6% inode=99%): [00:22:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:23:07] /sql on adenia is OK: DISK OK - free space: /sql 73777 MB (11% inode=99%): [00:39:18] /sql on adenia is WARNING: DISK WARNING - free space: /sql 68722 MB (10% inode=99%): [00:39:40] [[Special:Log/newusers]] create 10 * Blacksell * (New user account) [00:42:49] [[Contact]] ! 10https://wiki.toolserver.org/w/index.php?diff=6485&oldid=6406&rcid=8520 * Blacksell * (+1) (/* Specific users */ ) [00:45:15] [[Contact]] M 10https://wiki.toolserver.org/w/index.php?diff=6486&oldid=6485&rcid=8521 * Gifti * (-1) (Undo revision 6485 by [[Special:Contributions/Blacksell|Blacksell]] ([[User talk:Blacksell|talk]]): too specific ;)) [00:46:18] /sql on adenia is OK: DISK OK - free space: /sql 73517 MB (11% inode=99%): [00:56:18] /sql on adenia is WARNING: DISK WARNING - free space: /sql 68883 MB (10% inode=99%): [01:03:18] /sql on adenia is OK: DISK OK - free space: /sql 73078 MB (11% inode=99%): [01:07:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:09:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:12:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:15:50] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 201412 MB (3% inode=26%): [01:20:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 64952 MB (6% inode=99%): [01:22:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:08:27] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:10:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:12:27] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:15:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 201378 MB (3% inode=26%): [02:20:16] [[Getting started]] ! 10https://wiki.toolserver.org/w/index.php?diff=6487&oldid=6483&rcid=8522 * 71.57.249.133 * (+2201) (undo!) [02:20:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 64880 MB (6% inode=99%): [02:22:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:33:34] Is anyone available to give me a brief toolserver coaching lesson? [02:46:09] TParis: what do you need? [02:49:11] Sorry, modem got unplugged. [02:49:56] Betacommand: I'd like to know how to run a report off Wikipedia's database...or a copy of it. [02:50:14] what kind of report? [02:51:09] None in particular, I'd like to know how to do it is all. However, I am curious which article has the oldest top revision. [03:08:27] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:10:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:12:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:15:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 201244 MB (3% inode=26%): [03:19:10] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.039062/1.00, alarm hl:mem_free=13830.000000M/100M [03:21:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 64785 MB (6% inode=99%): [03:23:08] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:24:08] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [03:44:08] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.122070/1.00, alarm hl:mem_free=13360.000000M/100M [03:50:09] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [03:57:29] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.885742/2.00, alarm hl:mem_free=245.000000M/250M [03:58:29] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:03:18] Sun Grid Engine execd on nightshade is WARNING: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=1.179688/2.00, alarm hl:mem_free=153.000000M/250M [04:08:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:10:17] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [04:10:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:12:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:15:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 200612 MB (3% inode=26%): [04:22:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 64687 MB (6% inode=99%): [04:23:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:29:19] Load avg. on nightshade is WARNING: WARNING - load average: 16.92, 13.98, 11.49 [04:29:19] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.710938/1.50, alarm hl:mem_free=225.000000M/100M: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=1.420899/2.00, alarm hl:mem_free=225.000000M/250M [04:30:16] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6488&oldid=6456&rcid=8523 * 91.198.174.202 * (+0) (updated page) [04:33:18] Load avg. on nightshade is OK: OK - load average: 14.23, 14.70, 12.46 [04:37:18] Load avg. on nightshade is WARNING: WARNING - load average: 20.77, 17.51, 14.05 [04:42:10] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:42:10] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:42:10] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:42:10] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:42:58] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:42:59] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:42:59] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [04:42:59] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:09:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:10:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:12:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:15:29] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [05:15:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 201025 MB (3% inode=26%): [05:18:19] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.629883/1.50, alarm hl:mem_free=889.000000M/100M [05:23:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:23:20] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [05:23:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 64459 MB (6% inode=99%): [05:57:28] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=2.765137/1.50, alarm hl:mem_free=465.000000M/100M: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=2.266113/2.00, alarm hl:mem_free=465.000000M/250M [06:09:38] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:10:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:12:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:13:58] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:13:59] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:08] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:08] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:08] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:08] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:14:08] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:14:19] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:14:28] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:15:19] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3559 MB (99% inode=99%): [06:15:19] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [06:15:38] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [06:15:38] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [06:15:49] MySQL on z-dat-s3-a is OK: Uptime: 1438573 Threads: 12 Questions: 1429428863 Slow queries: 188010 Opens: 20440180 Flush tables: 2 Open tables: 16384 Queries per second avg: 993.643 [06:15:49] MySQL slave on z-dat-s3-a is OK: Uptime: 1438573 Threads: 12 Questions: 1429428864 Slow queries: 188010 Opens: 20440180 Flush tables: 2 Open tables: 16384 Queries per second avg: 993.643 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 194 [06:15:49] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [06:15:49] SMTP on z-dat-s7-a is OK: SMTP OK - 0.011 sec. response time [06:15:49] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:15:58] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:15:58] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:15:59] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:15:59] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:16:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 201727 MB (3% inode=26%): [06:20:29] Load avg. on nightshade is WARNING: WARNING - load average: 14.24, 13.91, 16.19 [06:23:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 64299 MB (6% inode=99%): [06:24:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:28:28] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 38.26, 21.31, 17.96 [06:33:29] Load avg. on nightshade is WARNING: WARNING - load average: 20.31, 22.97, 19.92 [06:37:30] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 24.77, 22.13, 20.09 [06:57:28] /sql on adenia is WARNING: DISK WARNING - free space: /sql 67500 MB (10% inode=99%): [06:57:29] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.888184/1.50, alarm hl:mem_free=678.000000M/100M: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=2.339355/2.00, alarm hl:mem_free=678.000000M/250M [07:02:38] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.611816/1.50, alarm hl:mem_free=754.000000M/100M [07:04:28] /sql on adenia is OK: DISK OK - free space: /sql 72053 MB (11% inode=99%): [07:04:39] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:05:28] Load avg. on nightshade is WARNING: WARNING - load average: 14.13, 12.45, 15.89 [07:05:28] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [07:07:28] Load avg. on nightshade is OK: OK - load average: 7.02, 10.37, 14.68 [07:09:39] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:10:48] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:16:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 200765 MB (3% inode=26%): [07:17:38] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 75575 MB (7% inode=99%): [07:22:38] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.505859/1.50, alarm hl:mem_free=1029.000000M/100M [07:23:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 64077 MB (6% inode=99%): [07:24:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:32:38] Load avg. on willow is WARNING: WARNING - load average: 15.90, 15.48, 12.54 [07:33:38] Load avg. on willow is OK: OK - load average: 12.64, 14.66, 12.45 [07:42:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:45:49] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.567871/1.50, alarm hl:mem_free=846.000000M/100M [07:46:49] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:02:49] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.536621/1.50, alarm hl:mem_free=856.000000M/100M [08:03:49] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:09:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:10:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:17:00] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 200563 MB (3% inode=26%): [08:23:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 63869 MB (6% inode=99%): [08:25:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:42:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:09:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:10:49] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:17:58] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 200383 MB (3% inode=26%): [09:23:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 63672 MB (6% inode=99%): [09:25:18] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:42:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:09:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:10:58] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:17:59] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 200366 MB (3% inode=26%): [10:23:49] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 63428 MB (6% inode=99%): [10:25:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:42:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:08:28] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.102539/1.00, alarm hl:mem_free=13370.000000M/100M [11:09:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:11:09] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:13:28] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [11:18:10] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 200203 MB (3% inode=26%): [11:19:28] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.299805/1.00, alarm hl:mem_free=13389.000000M/100M [11:23:57] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 63185 MB (6% inode=99%): [11:25:27] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:32:28] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [11:42:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:09:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:11:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:19:08] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199971 MB (3% inode=26%): [12:23:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 64086 MB (6% inode=99%): [12:25:30] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:42:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:44:39] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1960 [12:44:39] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1966.000000 [12:47:10] MySQL slave on thyme is OK: Uptime: 217514 Threads: 12 Questions: 56307082 Slow queries: 14965 Opens: 10572 Flush tables: 1 Open tables: 2267 Queries per second avg: 258.866 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1666 [12:47:10] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1660.000000 [13:09:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:12:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:19:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199916 MB (3% inode=26%): [13:23:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 63730 MB (6% inode=99%): [13:26:28] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:27:39] Sun Grid Engine execd on nightshade is WARNING: longrun@nightshade exceedes load threshold: alarm hl:np_load_long=0.882324/2.00, alarm hl:mem_free=138.000000M/250M [13:29:29] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.189453/1.00, alarm hl:mem_free=13112.000000M/100M [13:29:40] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [13:39:28] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [13:45:28] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.083008/1.00, alarm hl:mem_free=14046.000000M/100M [14:10:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:12:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:13:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:19:20] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199894 MB (3% inode=26%): [14:23:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 63522 MB (6% inode=99%): [14:26:28] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:10:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:12:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:13:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:19:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199631 MB (3% inode=26%): [15:23:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 63310 MB (6% inode=99%): [15:26:29] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:41:28] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.175781/1.00, alarm hl:mem_free=13974.000000M/100M [15:52:27] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [15:54:38] /sql on adenia is WARNING: DISK WARNING - free space: /sql 67718 MB (10% inode=99%): [16:01:38] /sql on adenia is OK: DISK OK - free space: /sql 72030 MB (11% inode=99%): [16:09:05] hello [16:11:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:14:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:20:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199611 MB (3% inode=26%): [16:23:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 63089 MB (6% inode=99%): [16:26:39] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:42:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:46:23] [[Special:Log/newusers]] create 10 * ZeaForUs * (New user account) [17:11:09] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:14:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:17:58] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.709473/1.50, alarm hl:mem_free=722.000000M/100M [17:18:57] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [17:20:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199264 MB (3% inode=26%): [17:24:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62854 MB (6% inode=99%): [17:26:39] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:42:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:11:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:14:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:20:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199048 MB (3% inode=25%): [18:25:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62585 MB (6% inode=99%): [18:26:39] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:40:45] hi all [18:42:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:50:59] who has to accept the account requests? [18:52:12] * thekla points to DaBPunkt ;) [18:52:32] nickanc: I will look into it later tonight [18:54:06] no problem, DabPunkt. :) [18:54:14] I was only wondering if it was all right with my application. [18:54:37] nickanc: do you have the number at hand? [18:55:08] daBPunkt https://jira.toolserver.org/browse/ACCAPP-432 [18:56:21] nickanc: please add a link to the approval on the bot [18:58:53] daBPunkt i just added a comment [18:59:22] 3(commented) [ACCAPP-432] Bot account on TS <10https://jira.toolserver.org/browse/ACCAPP-432> (Nickanc) [19:11:29] DaBPunkt: Hi. I'm working on the logging stuff. You told me I should write logs to /var/log/tomcat/... - but where would it go to? on which server? on willow there is no /var/log/tomcat directory - or at least it's not visible from my account. [19:12:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:14:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:18:08] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 69334 MB (7% inode=99%): [19:20:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199263 MB (3% inode=26%): [19:25:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62369 MB (6% inode=99%): [19:26:40] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:42:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:12:29] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:14:29] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:21:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199184 MB (3% inode=25%): [20:25:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 62153 MB (6% inode=99%): [20:26:40] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:37:44] [[User:Iluvatar]] !N 10https://wiki.toolserver.org/w/index.php?oldid=6489&rcid=8525 * Iluvatar * (+113) (Created page with "* Account on [[:ru:User:Iluvatar|Russian Wikipedia]] * Account on toolserver http://tools.wikimedia.de/~iluvatar/") [20:42:10] [[List of Wikimedia bots]] ! 10https://wiki.toolserver.org/w/index.php?diff=6490&oldid=6024&rcid=8526 * Iluvatar * (+118) (/* {{#language:ru}} */ +1) [20:42:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:42:41] jongleur1: It will be avaiable on each webserver [20:43:15] DaBPunkt: so I have to log in into a webserver from willow first, right? [20:44:00] either that or login to a webserver directly [20:44:19] k, thanks [20:57:59] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_long=0.708496/2.00, alarm hl:mem_free=234.000000M/250M [20:58:58] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [21:02:49] Load avg. on hyacinth is WARNING: WARNING - load average: 19.32, 13.79, 10.05 [21:12:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:15:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:21:28] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 198739 MB (3% inode=25%): [21:25:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 61936 MB (6% inode=99%): [21:26:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:38:15] DaBPunkt: committing currently. Logging now goes to /var/log/tomcat/lalm.log (+ logrotate for max 10 files with 200KB each) [21:46:58] Load avg. on hyacinth is OK: OK - load average: 10.37, 13.04, 14.95 [21:48:08] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.747070/1.50, alarm hl:mem_free=651.000000M/100M [21:50:01] DaBPunkt: do you have time to help me again? (of course not, I know...) I published an updated .war file as lalm.war.live around 20 mintes ago. [21:50:21] SEVERE: Error deploying web application archive lalm.war [21:50:23] java.lang.NoClassDefFoundError: org/apache/xerces/util/EncodingMap [21:50:35] And I have always time for my users ;) [21:50:41] ;) that's great [21:51:48] so probably it's crashing before initializing the logger... the application should do logging now to /var/log/tomcat/lalm.log, but that file is not existing yet [21:55:09] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [22:01:13] and next try [22:01:27] .war.live is updated [22:06:12] same error [22:06:28] damn... [22:06:29] thx [22:12:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [22:12:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:15:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:21:28] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 198829 MB (3% inode=25%): [22:25:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 61668 MB (6% inode=99%): [22:27:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:42:59] Load avg. on hyacinth is WARNING: WARNING - load average: 15.98, 14.39, 13.34 [22:43:59] Load avg. on hyacinth is OK: OK - load average: 14.73, 14.36, 13.40 [22:45:28] DaBPunkt: Anythning on https://jira.toolserver.org/browse/TS-1220 ? [22:45:46] Users are enable to change their interface language from many tools as it goes through this tool (Toolserver Intuitoin) [22:45:56] the form save on that page keeps giving 502 Bad Gateway [22:46:10] http://toolserver.org/~intuition/ [22:46:25] works locally and no php error that I can see [22:46:27] no database involved [22:50:19] Krinkle: "Dec 28 22:49:21 wolfsbane php-cgi[24695]: [ID 702911 user.notice] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 89 bytes) in /home/project/i/n/t/intuition/ToolserverI18N/TsIntuition.php on line 787" [22:56:28] Hm.. [22:56:29] thanks :) [22:57:47] Krinkle: a tipp: use http://wolfsbane.toolserver.org/~intuition/index.php?action=renewcookies so that you will know on which server the code is executed; then you can look in the logs easier [22:58:05] yeah [22:58:09] (it show BTW that this is not a ngix-problem ;)) [22:58:21] So, bold question, can I raise memory limit ? [22:58:56] no idea, but do you think it is a good idea to use more then 130MB of RAM for a webpage? [22:59:02] this amount of memory isn't normally used by the framework. Only on the preferences form where it needs all languages [22:59:07] 130MB ? [22:59:12] Eh.. oops [22:59:15] I wonder how that happens though [22:59:21] that's way too much [22:59:52] 134.217.728 bytes are 134MB (1.000.000Bytes=1MB), AFAIS [23:00:23] DaBPunkt: next try. :( [23:00:38] but at least no log file yet... [23:00:51] DaBPunkt: I wonder where all this memory is being used for. Any idea ? [23:01:32] jongleur1: still the same error. I guess the easiest solution is to just include the damn class/jar ;) [23:02:07] DaBPunkt: that's the third jar archive containing that class I tried now. [23:02:11] Krinkle: check for arrays, loops and that stuff of things [23:02:30] Krinkle: and downloaded files etc. ;) [23:02:37] DaBPunkt: Well, it can't be the files [23:03:49] jongleur1: http://xerces.apache.org/xerces2-j/ looks good for me [23:04:35] how can you use something in your IDE, but not knowing what jar it is in? [23:05:48] Krinkle: I bet there is a memory-hole somewhere in the code [23:05:54] DaBPunkt: I use Jetty as a servlet container here. Jetty provides more java classes than Tomcat out of the box - do you know every class the base java runtime includes? ;) [23:06:00] DaBPunkt: I think I found something that is heavy [23:06:32] jongleur1: of course not :-). [23:06:35] DaBPunkt: In the preferences I need to know all used languages, so I scan the directory and then call parseTextdomainByFile( $path ), however [23:06:43] DaBPunkt: that does more then just index languages, it also loads all messages [23:06:48] which is not needed and is rather expensive [23:12:38] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:15:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:18:01] MySQL slave on rosemary is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:18:01] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:18:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:18:08] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:18:09] MySQL on z-dat-s3-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:18:09] MySQL slave on z-dat-s3-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:18:28] MySQL on rosemary is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:18:49] MySQL on thyme is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:18:50] MySQL slave on thyme is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:18:59] MySQL slave on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:19:08] MySQL on cassia is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:19:38] MySQL slave on z-dat-s7-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:19:38] MySQL on z-dat-s4-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:19:49] MySQL slave on cassia is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:19:50] s5 replag on cassia is CRITICAL: QUERY CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:20:00] MySQL on z-dat-s7-a is CRITICAL: Access denied for user nagios@damiana-bge0.esi.toolserver.org (using password: YES) [23:21:42] DaBPunkt: where is that error log ? I'm on wolfsbane root [23:21:53] /var/log/apache not I guess [23:22:04] I tried some other errorl-ish files but no look [23:22:07] luck* [23:22:28] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 196989 MB (3% inode=25%): [23:23:44] I do see the request in /var/log/http/access [23:23:52] (tail -n 100 | grep intuition) [23:25:08] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 61441 MB (6% inode=99%): [23:27:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:30:32] re [23:30:56] Krinkle: php-error are logged to /var/log/userlog [23:31:01] Thx [23:32:12] (how https://wiki.toolserver.org/view/Web_hosting#Logs would had told you, but it is easier to ask here ;-)) [23:32:46] Thx, I did look on the wiki, but missed that [23:32:49] some people should be punished for filling it with errors :P [23:33:10] Platonides: yes. I plan to write a programm that will send evil mails ;) [23:33:43] do you want help with that? [23:33:53] seems like a simple script [23:34:03] (maybe we can endow a "log-polluter of the week"-price too ;)) [23:34:34] "Most errors were produced by X tools" [23:34:57] * jongleur1 hates his slow netbook... 4 minutes build time for a small project :/ [23:35:19] Gives more time for the swordfights, though, jongleurl [23:35:28] Every cloud has a silver lining. [23:35:31] hehe [23:35:39] *g* [23:36:43] ;) [23:41:50] next commit... *bored* [23:42:34] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:45:45] DaBPunkt: next try [23:45:53] .war.live is online [23:49:12] DaBPunkt: Aaaaaargh! [23:49:19] I had an infinite loop between two functions [23:50:23] the functions were fine but they were passing a constant, which were supposed to be two different constants [23:50:38] but in Defines.php turns out both DO_TRACK and NO_TRACK were set to "true" [23:50:39] xD [23:51:22] DaBPunkt: next try? [23:54:02] Krinkle: so I won my bet :) [23:54:46] jongleur1: still the same error. Are you sure that the class is not already in a jar of my list from yesterday? [23:55:32] DaBPunkt: Sure, no doubt a about it [23:55:48] Krinkle, looks like made by some greedy company ;) [23:59:03] DaBPunkt: Not completely sure, but nothing of that looks like sth that could contain that class [23:59:35] 3(resolved) [TS-1220] "502 Bad Gateway nginx/1.0.4" when using HTML Form with POST <10https://jira.toolserver.org/browse/TS-1220> (Krinkle)