[00:03:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [00:05:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54658 MB (5% inode=99%): [00:19:21] nacht ts [00:49:14] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:51:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:55:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:03:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:05:21] Will all scheduled cron jobs be cancelled, ors omething? [01:05:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53972 MB (5% inode=99%): [01:49:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:51:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:55:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:03:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:05:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53838 MB (5% inode=99%): [02:17:34] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:52] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:18:02] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:18:33] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 3490 MB (99% inode=99%): [02:18:33] SMTP on z-dat-s7-a is OK: SMTP OK - 7.691 sec. response time [02:18:42] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:18:51] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [02:18:51] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [02:19:02] MySQL on z-dat-s3-a is OK: Uptime: 3238766 Threads: 15 Questions: 3495840803 Slow queries: 294200 Opens: 44214767 Flush tables: 2 Open tables: 16384 Queries per second avg: 1079.374 [02:19:03] MySQL slave on z-dat-s3-a is OK: Uptime: 3238766 Threads: 15 Questions: 3495840803 Slow queries: 294200 Opens: 44214767 Flush tables: 2 Open tables: 16384 Queries per second avg: 1079.374 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 182 [02:47:53] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:53] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:53] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:53] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:53] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:54] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:03] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [02:49:03] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [02:49:24] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [02:49:24] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [02:49:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:49:24] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 93.91, 37.80, 18.46 [02:49:32] MySQL on z-dat-s3-a is OK: Uptime: 3240598 Threads: 10 Questions: 3498723468 Slow queries: 294380 Opens: 44262762 Flush tables: 2 Open tables: 16384 Queries per second avg: 1079.653 [02:49:32] MySQL slave on z-dat-s3-a is OK: Uptime: 3240598 Threads: 10 Questions: 3498723469 Slow queries: 294380 Opens: 44262762 Flush tables: 2 Open tables: 16384 Queries per second avg: 1079.653 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 248 [02:49:33] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=12.245605/1.75, alarm hl:np_load_avg=4.989258/2.00, alarm hl:mem_free=1820.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=12.245605/1.50, alarm hl:np_load_long=2.414551/1.75, alarm hl:mem_free=1820.000000M/250M [02:49:33] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:49:42] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:49:42] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:49:42] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:49:42] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:49:42] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:49:42] MySQL on z-dat-s6-a is OK: Uptime: 4507530 Threads: 8 Questions: 857242833 Slow queries: 300756 Opens: 8641183 Flush tables: 2 Open tables: 2851 Queries per second avg: 190.180 [02:49:42] MySQL slave on z-dat-s6-a is OK: Uptime: 4507530 Threads: 7 Questions: 857242832 Slow queries: 300756 Opens: 8641183 Flush tables: 2 Open tables: 2851 Queries per second avg: 190.180 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 125 [02:52:22] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:56:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:03:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:06:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53644 MB (5% inode=99%): [03:49:42] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=25.896973/1.75, alarm hl:np_load_avg=22.887207/2.00, alarm hl:mem_free=1807.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=25.896973/1.50, alarm hl:np_load_long=18.467285/1.75, alarm hl:mem_free=1807.000000M/250M [03:50:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:50:23] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 193.10, 182.30, 149.48 [03:52:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:56:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:03:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:06:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53484 MB (5% inode=99%): [04:12:20] 3(commented) [TS-1263] Please change my SSH public key <10https://jira.toolserver.org/browse/TS-1263> (Junaid PV) [04:30:16] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=6551&oldid=6538&rcid=8617 * 91.198.174.202 * (+5) (updated page) [04:49:43] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=22.259277/1.75, alarm hl:np_load_avg=23.169922/2.00, alarm hl:mem_free=1698.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=22.259277/1.50, alarm hl:np_load_long=20.951172/1.75, alarm hl:mem_free=1698.000000M/250M [04:50:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:50:25] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 237.97, 193.80, 171.27 [04:52:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:56:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:03:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54254 MB (5% inode=99%): [05:49:44] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=24.665527/1.75, alarm hl:np_load_avg=27.064453/2.00, alarm hl:mem_free=1826.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=24.665527/1.50, alarm hl:np_load_long=28.876953/1.75, alarm hl:mem_free=1826.000000M/250M [05:50:24] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:50:24] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 141.09, 198.55, 223.88 [05:52:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:56:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:03:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:06:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54170 MB (5% inode=99%): [06:18:36] does anybody know what BlackoutWelcomer.py is? matthewr is using nearly all power of nightshade [06:49:53] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=25.216797/1.75, alarm hl:np_load_avg=29.801758/2.00, alarm hl:mem_free=1672.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=25.216797/1.50, alarm hl:np_load_long=30.028809/1.75, alarm hl:mem_free=1672.000000M/250M [06:50:33] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 250.27, 250.80, 244.80 [06:50:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:52:34] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:57:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:02:34] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.915039/1.75, alarm hl:np_load_avg=1.460938/2.00, alarm hl:mem_free=1813.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.915039/1.50, alarm hl:np_load_long=1.148438/1.75, alarm hl:mem_free=1813.000000M/250M [07:04:32] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:07:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54094 MB (5% inode=99%): [07:20:07] Sumana Harihareswara * Re: [Toolserver-l] Editing en.wp via API to be disabled on 18 January [07:27:32] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.596191/1.50, alarm hl:np_load_long=1.254883/1.75, alarm hl:mem_free=1650.000000M/250M [07:28:31] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:33:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:50:23] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=32.388184/1.75, alarm hl:np_load_avg=33.370606/2.00, alarm hl:mem_free=1487.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=32.388184/1.50, alarm hl:np_load_long=32.496094/1.75, alarm hl:mem_free=1487.000000M/250M [07:50:44] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:51:33] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 228.15, 257.43, 257.29 [07:51:51] @replag [07:51:51] Chris_G: s1-pri: 52s [-0.03 s/s]; s1-sec: 52s [+0.00 s/s]; s2-pri: 20s [+0.00 s/s]; s3-rr: 29s [+0.00 s/s]; s3-user: 29s [+0.00 s/s] [07:52:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:56:08] Lars Aronsson * Re: [Toolserver-l] interwiki.py [07:58:23] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:07:24] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 45741 MB (4% inode=99%): [08:08:23] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53970 MB (5% inode=99%): [08:26:21] 3(updated) [TS-1264] please extend account <10https://jira.toolserver.org/browse/TS-1264> (Devunt) [08:26:22] 3(created) [TS-1264] please extend account; Toolserver; Task <10https://jira.toolserver.org/browse/TS-1264> (Devunt) [08:33:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [08:50:32] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=6.281738/1.75, alarm hl:np_load_avg=7.415527/2.00, alarm hl:mem_free=2234.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=6.281738/1.50, alarm hl:np_load_long=14.302734/1.75, alarm hl:mem_free=2234.000000M/250M [08:51:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:52:13] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 50.17, 56.88, 106.41 [08:53:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:58:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:09:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53835 MB (5% inode=99%): [09:50:32] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=14.624023/1.75, alarm hl:np_load_avg=14.828613/2.00, alarm hl:mem_free=1330.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=14.624023/1.50, alarm hl:np_load_long=12.676270/1.75, alarm hl:mem_free=1330.000000M/250M [09:51:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:52:14] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 121.06, 121.09, 104.59 [09:53:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:59:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:03:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:10:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53811 MB (5% inode=99%): [10:24:13] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1928.000000 [10:24:13] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1933.000000 [10:28:12] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 10.000000 [10:28:12] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 14.000000 [10:50:42] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=14.767090/1.75, alarm hl:np_load_avg=15.454102/2.00, alarm hl:mem_free=1636.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=14.767090/1.50, alarm hl:np_load_long=15.295410/1.75, alarm hl:mem_free=1636.000000M/250M [10:51:14] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:51:20] 3(created) [DBQ-173] problem with fr.wiki data base; Database Queries; Task <10https://jira.toolserver.org/browse/DBQ-173> (reza) [10:52:13] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 118.84, 122.47, 122.06 [10:53:14] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:59:31] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:10:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53793 MB (5% inode=99%): [11:12:52] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:13:32] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:19:22] 3(commented) [DBQ-173] problem with fr.wiki data base <10https://jira.toolserver.org/browse/DBQ-173> (reza) [11:19:23] 3(updated) [DBQ-173] problem with fr.wiki data base <10https://jira.toolserver.org/browse/DBQ-173> (reza) [11:21:22] 3(commented) [DBQ-173] problem with fr.wiki data base <10https://jira.toolserver.org/browse/DBQ-173> (reza) [11:33:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:35:03] Dennis Tobar * Re: [Toolserver-l] interwiki.py [11:37:51] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:38:12] /v/sql on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:38:12] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:38:12] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:38:23] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:38:33] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [11:38:41] /v/sql on hyacinth is OK: DISK OK - free space: /v/sql 215649 MB (22% inode=99%): [11:38:41] /tmp on hyacinth is OK: DISK OK - free space: /tmp 3682 MB (99% inode=99%): [11:38:51] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [11:39:12] SMTP on z-dat-s4-a is OK: SMTP OK - 0.049 sec. response time [11:51:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:51:41] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=16.141113/1.75, alarm hl:np_load_avg=15.763184/2.00, alarm hl:mem_free=1797.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=16.141113/1.50, alarm hl:np_load_long=15.617188/1.75, alarm hl:mem_free=1797.000000M/250M [11:53:11] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 123.32, 124.78, 124.55 [11:53:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:59:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:10:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53765 MB (5% inode=99%): [12:33:24] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:51:42] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=14.992676/1.75, alarm hl:np_load_avg=15.264648/2.00, alarm hl:mem_free=1757.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=14.992676/1.50, alarm hl:np_load_long=15.418945/1.75, alarm hl:mem_free=1757.000000M/250M [12:52:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:53:12] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 115.59, 119.43, 122.17 [12:53:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:59:41] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:10:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53742 MB (5% inode=99%): [13:21:21] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1927.000000 [13:22:18] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1976.000000 [13:37:15] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 24.000000 [13:37:38] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 44.000000 [13:51:47] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=14.908691/1.75, alarm hl:np_load_avg=14.565430/2.00, alarm hl:mem_free=1650.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=14.908691/1.50, alarm hl:np_load_long=14.561035/1.75, alarm hl:mem_free=1650.000000M/250M [13:52:36] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:53:16] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 117.85, 116.54, 116.48 [13:53:37] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:59:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:03:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:10:36] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53714 MB (5% inode=99%): [14:51:57] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=13.011719/1.75, alarm hl:np_load_avg=13.222168/2.00, alarm hl:mem_free=2463.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=13.011719/1.50, alarm hl:np_load_long=13.371582/1.75, alarm hl:mem_free=2463.000000M/250M [14:52:37] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:53:40] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:54:15] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 109.46, 107.08, 107.29 [15:00:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:03:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:10:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53689 MB (5% inode=99%): [15:43:15] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1962.000000 [15:43:38] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1981.000000 [15:50:15] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 6.000000 [15:50:36] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 24.000000 [15:52:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:52:46] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=13.333984/1.75, alarm hl:np_load_avg=13.381348/2.00, alarm hl:mem_free=2519.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=13.333984/1.50, alarm hl:np_load_long=13.309570/1.75, alarm hl:mem_free=2519.000000M/250M [15:54:17] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 97.78, 103.46, 105.17 [15:54:36] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:01:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:03:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:10:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53653 MB (5% inode=99%): [16:22:16] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1927.000000 [16:22:35] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1947.000000 [16:50:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [16:50:36] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3627.000000 [16:52:39] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:52:56] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=12.238770/1.75, alarm hl:np_load_avg=11.663574/2.00, alarm hl:mem_free=2474.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=12.238770/1.50, alarm hl:np_load_long=12.331055/1.75, alarm hl:mem_free=2474.000000M/250M [16:54:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:55:15] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 87.41, 89.97, 96.56 [17:01:57] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:03:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [17:11:36] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53627 MB (5% inode=99%): [17:14:36] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 10.000000 [17:15:15] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 50.000000 [17:37:34] hi all. [17:38:13] my connection to willow is extremely slow. are there any problems with the server? [17:41:39] is the toolserver intentionally redirected to the wmf blog? [17:47:03] Thehelpfulone: he is redirecting? Not to me [17:47:47] Alchimista: the ACC tool is [17:48:06] Thehelpfulone: link [17:48:18] http://toolserver.org/~acc/acc.php [17:48:21] it's provably an action by the owner, ts wiki and site are ok [17:48:40] Thehelpfulone: http://toolserver.org/~alchimista/stats.php [17:48:47] oh okay [17:48:54] * Thehelpfulone didn't see any svn changes [17:49:49] :P [17:52:56] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=12.308594/1.75, alarm hl:np_load_avg=12.402832/2.00, alarm hl:mem_free=2428.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=12.308594/1.50, alarm hl:np_load_long=13.584961/1.75, alarm hl:mem_free=2428.000000M/250M [17:53:38] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:54:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:55:16] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 102.68, 100.50, 107.84 [17:59:15] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1933.000000 [17:59:36] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1955.000000 [18:03:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:03:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:03:37] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 10.000000 [18:04:17] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 49.000000 [18:11:36] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53608 MB (5% inode=99%): [18:51:41] zzz =__= [18:53:56] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=15.039551/1.75, alarm hl:np_load_avg=15.871582/2.00, alarm hl:mem_free=1891.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=15.039551/1.50, alarm hl:np_load_long=14.979492/1.75, alarm hl:mem_free=1891.000000M/250M [18:54:37] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:55:15] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 120.57, 124.86, 119.65 [18:55:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:03:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [19:03:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:11:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53580 MB (5% inode=99%): [19:54:05] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=13.294434/1.75, alarm hl:np_load_avg=14.041016/2.00, alarm hl:mem_free=2358.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=13.294434/1.50, alarm hl:np_load_long=14.471680/1.75, alarm hl:mem_free=2358.000000M/250M [19:54:45] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:55:16] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 118.09, 114.19, 116.12 [19:55:47] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:03:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:04:56] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:07:36] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 44186 MB (4% inode=99%): [20:12:36] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55563 MB (5% inode=99%): [20:13:17] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1947.000000 [20:13:36] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1967.000000 [20:18:44] @replag [20:18:45] nosy: s1-pri: 37m 59s [+0.05 s/s]; s1-sec: 37m 59s [+0.05 s/s]; s3-rr: 27s [-0.00 s/s]; s3-user: 27s [-0.00 s/s] [20:20:36] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 9.000000 [20:21:15] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 49.000000 [20:52:36] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1929.000000 [20:53:14] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1970.000000 [20:54:05] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=16.096191/1.75, alarm hl:np_load_avg=15.918457/2.00, alarm hl:mem_free=2612.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=16.096191/1.50, alarm hl:np_load_long=15.791504/1.75, alarm hl:mem_free=2612.000000M/250M [20:54:45] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:55:16] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 134.44, 129.10, 127.04 [20:56:45] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:03:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [21:04:56] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:12:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55546 MB (5% inode=99%): [21:13:25] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 24.000000 [21:13:36] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 41.000000 [21:54:15] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=14.441895/1.75, alarm hl:np_load_avg=14.351074/2.00, alarm hl:mem_free=2442.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=14.441895/1.50, alarm hl:np_load_long=14.819824/1.75, alarm hl:mem_free=2442.000000M/250M [21:54:46] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:55:24] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 121.59, 116.57, 118.84 [21:56:45] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:03:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [22:04:56] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:12:36] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55528 MB (5% inode=99%): [22:30:36] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1925.000000 [22:31:25] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1969.000000 [22:54:16] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=14.594727/1.75, alarm hl:np_load_avg=14.568359/2.00, alarm hl:mem_free=2402.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=14.594727/1.50, alarm hl:np_load_long=14.728516/1.75, alarm hl:mem_free=2402.000000M/250M [22:54:46] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:55:26] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 125.59, 118.95, 118.56 [22:56:46] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:58:35] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3605.000000 [22:59:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3649.000000 [23:03:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:05:09] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:12:40] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55511 MB (5% inode=99%): [23:52:20] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [23:52:41] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [23:54:21] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=14.355469/1.75, alarm hl:np_load_avg=14.319336/2.00, alarm hl:mem_free=2410.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=14.355469/1.50, alarm hl:np_load_long=14.528320/1.75, alarm hl:mem_free=2410.000000M/250M [23:54:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:56:20] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 119.96, 117.42, 117.13 [23:56:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:57:22] MySQL slave on rosemary is OK: Uptime: 6069096 Threads: 15 Questions: 1732534899 Slow queries: 749839 Opens: 9435 Flush tables: 1 Open tables: 889 Queries per second avg: 285.468 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [23:57:41] MySQL slave on thyme is OK: Uptime: 1020431 Threads: 5 Questions: 318151708 Slow queries: 131905 Opens: 59891 Flush tables: 1 Open tables: 2577 Queries per second avg: 311.781 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [23:57:41] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 16.000000 [23:58:21] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 58.000000