[00:01:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41311.000000 [00:01:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81553.000000 [00:02:33] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.991211/1.75, alarm hl:np_load_avg=0.893066/2.00, alarm hl:mem_free=258.000000M/300M [00:02:54] MySQL slave on z-dat-s7-a is OK: Uptime: 2780633 Threads: 8 Questions: 663990513 Slow queries: 114624 Opens: 7961937 Flush tables: 1 Open tables: 3892 Queries per second avg: 238.791 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [00:02:54] MySQL on z-dat-s7-a is OK: Uptime: 2780633 Threads: 7 Questions: 663990515 Slow queries: 114624 Opens: 7961937 Flush tables: 1 Open tables: 3892 Queries per second avg: 238.791 [00:06:33] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:07:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 39891 [00:09:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 80412 [00:10:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:11:21] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47603 MB (4% inode=99%): [00:13:24] 3(created) [MNT-1177] Fixed mysql-access for naios on s7; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1177> (DaB.) [00:13:25] 3(updated) [MNT-1177] Fixed mysql-access for naios on s7 <10https://jira.toolserver.org/browse/MNT-1177> (DaB.) [00:13:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:17:32] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:35:06] nacht ts [00:49:33] [[Query service]] 10https://wiki.toolserver.org/w/index.php?diff=6585&oldid=6584&rcid=8666 * MZMcBride * (-35) (rv) [00:50:43] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:70.40.185.9910]] with an expiry time of 1 week (account creation disabled): inappropriate behavior) [00:56:53] / on willow is WARNING: DISK WARNING - free space: / 22903 MB (20% inode=99%): [01:01:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31764.000000 [01:01:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71749.000000 [01:07:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30734 [01:09:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 70194 [01:10:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:11:22] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48365 MB (4% inode=99%): [01:13:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:17:33] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:42:33] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.418945/1.75, alarm hl:np_load_avg=0.458496/2.00, alarm hl:mem_free=222.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.418945/1.50, alarm hl:np_load_long=0.491211/1.75, alarm hl:mem_free=222.000000M/250M [01:45:32] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [01:48:32] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.430176/1.75, alarm hl:np_load_avg=0.461914/2.00, alarm hl:mem_free=280.000000M/300M [01:56:53] / on willow is WARNING: DISK WARNING - free space: / 22458 MB (20% inode=99%): [02:00:42] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [02:01:34] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65002.000000 [02:02:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24799.000000 [02:08:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24460 [02:09:13] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64537 [02:10:32] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:11:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48223 MB (4% inode=99%): [02:13:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:17:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:33:24] If anyone sees a "root" login attempt, my bad, habit [02:56:52] / on willow is WARNING: DISK WARNING - free space: / 22337 MB (20% inode=99%): [03:01:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54593.000000 [03:02:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15972.000000 [03:08:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14615 [03:10:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52611 [03:10:33] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:10:33] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:10:43] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:10:43] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:10:43] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:10:43] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:11:13] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [03:11:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48061 MB (4% inode=99%): [03:11:33] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:11:33] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:11:33] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:11:33] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:13:53] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:17:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:56:53] / on willow is WARNING: DISK WARNING - free space: / 22156 MB (19% inode=99%): [04:01:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43731.000000 [04:02:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5991.000000 [04:02:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.614746/1.75, alarm hl:np_load_avg=0.576172/2.00, alarm hl:mem_free=294.000000M/300M [04:03:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [04:09:22] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4891 [04:10:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 43051 [04:10:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:12:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48831 MB (5% inode=99%): [04:13:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.592773/1.75, alarm hl:np_load_avg=0.530273/2.00, alarm hl:mem_free=230.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.592773/1.50, alarm hl:np_load_long=0.499512/1.75, alarm hl:mem_free=230.000000M/250M [04:14:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:18:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:30:03] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3449.000000 [04:30:22] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3394 [04:45:14] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1638.000000 [04:45:23] MySQL slave on thyme is OK: Uptime: 1728898 Threads: 7 Questions: 621850271 Slow queries: 298568 Opens: 91569 Flush tables: 1 Open tables: 2653 Queries per second avg: 359.680 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1572 [04:56:53] / on willow is WARNING: DISK WARNING - free space: / 22105 MB (19% inode=99%): [05:01:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36666.000000 [05:10:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36092 [05:11:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:13:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48704 MB (4% inode=99%): [05:13:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.430176/1.75, alarm hl:np_load_avg=0.420410/2.00, alarm hl:mem_free=232.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.430176/1.50, alarm hl:np_load_long=0.427246/1.75, alarm hl:mem_free=232.000000M/250M [05:14:43] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [05:15:04] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:18:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:25:33] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:25:33] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:25:43] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:25:43] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:26:22] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:26:22] SMTP on z-dat-s7-a is OK: SMTP OK - 0.045 sec. response time [05:26:32] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:26:32] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:47:32] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:32] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:33] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:33] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:47:43] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:44] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:44] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:47:44] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:43] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [05:48:43] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [05:49:02] MySQL on z-dat-s7-a is OK: Uptime: 2801409 Threads: 7 Questions: 668847578 Slow queries: 115469 Opens: 8021010 Flush tables: 1 Open tables: 3960 Queries per second avg: 238.753 [05:49:02] MySQL slave on z-dat-s7-a is OK: Uptime: 2801409 Threads: 8 Questions: 668847578 Slow queries: 115469 Opens: 8021010 Flush tables: 1 Open tables: 3960 Queries per second avg: 238.753 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 269 [05:49:12] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [05:49:22] SMTP on z-dat-s3-a is OK: SMTP OK - 0.026 sec. response time [05:49:22] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:49:22] SMTP on hyacinth is OK: SMTP OK - 0.003 sec. response time [05:49:32] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:49:33] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:49:33] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [05:49:33] SMTP on z-dat-s6-a is OK: SMTP OK - 0.003 sec. response time [05:57:53] / on willow is WARNING: DISK WARNING - free space: / 22047 MB (19% inode=99%): [06:01:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31958.000000 [06:10:12] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30892 [06:11:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:13:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49503 MB (5% inode=99%): [06:14:53] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:15:42] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:16:03] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:19:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:23:33] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:23:43] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:24:23] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [06:24:24] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [06:24:24] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [06:24:32] MySQL on z-dat-s7-a is OK: Uptime: 2803533 Threads: 7 Questions: 669263730 Slow queries: 115589 Opens: 8035582 Flush tables: 1 Open tables: 3975 Queries per second avg: 238.721 [06:24:33] MySQL slave on z-dat-s7-a is OK: Uptime: 2803533 Threads: 7 Questions: 669263766 Slow queries: 115589 Opens: 8035616 Flush tables: 1 Open tables: 3975 Queries per second avg: 238.721 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 245 [06:24:33] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [06:58:52] / on willow is WARNING: DISK WARNING - free space: / 21991 MB (19% inode=99%): [07:01:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22196.000000 [07:10:22] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20620 [07:12:43] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:12:43] Sun Grid Engine execd on willow is WARNING: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.530274/1.50, alarm hl:np_load_long=0.814453/1.75, alarm hl:mem_free=365.000000M/250M [07:13:52] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [07:14:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49333 MB (5% inode=99%): [07:16:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:19:42] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:58:52] / on willow is WARNING: DISK WARNING - free space: / 21937 MB (19% inode=99%): [08:01:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8808.000000 [08:10:23] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6399 [08:11:14] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 29557 MB (3% inode=99%): [08:12:44] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:15:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50169 MB (5% inode=99%): [08:16:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:17:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.564453/1.75, alarm hl:np_load_avg=0.640137/2.00, alarm hl:mem_free=243.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=0.564453/1.50, alarm hl:np_load_long=0.663574/1.75, alarm hl:mem_free=243.000000M/250M [08:19:44] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:19:44] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [08:20:23] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3442 [08:20:44] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3357.000000 [08:24:23] MySQL slave on rosemary is OK: Uptime: 6790716 Threads: 26 Questions: 2000556021 Slow queries: 841909 Opens: 10182 Flush tables: 1 Open tables: 900 Queries per second avg: 294.601 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1786 [08:24:43] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1690.000000 [08:55:43] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.281250/1.75, alarm hl:np_load_avg=0.721680/2.00, alarm hl:mem_free=235.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.281250/1.50, alarm hl:np_load_long=0.587402/1.75, alarm hl:mem_free=235.000000M/250M [08:56:52] / on willow is OK: DISK OK - free space: / 26562 MB (23% inode=99%): [09:05:52] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [09:08:52] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.619629/1.75, alarm hl:np_load_avg=0.743652/2.00, alarm hl:mem_free=265.000000M/300M [09:13:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:15:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50042 MB (5% inode=99%): [09:16:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:19:44] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:33:37] @replag all [09:33:38] nosy: s1-pri: 1s [-1.31 s/s]; s1-sec: 1s [-2.35 s/s]; s1-sec-c: 2s [+0.00 s/s]; s2-pri: 1s [-0.00 s/s]; s2/s5-pri-c: 2s [-0.00 s/s]; s3-rr: 17s [-0.00 s/s]; s3-user: 17s [-0.00 s/s]; s4-rr: 2s [+0.00 s/s] [09:33:39] nosy: s4-user: 2s [-0.00 s/s]; s5-rr: 2s [-0.00 s/s]; s5-user: 2s [-0.00 s/s]; s6-rr: 5s [-0.01 s/s]; s6-user: 5s [-0.01 s/s]; s7-rr: 6s [-0.00 s/s]; s7-user: 6s [-0.00 s/s] [09:57:52] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.599609/1.75, alarm hl:np_load_avg=0.528320/2.00, alarm hl:mem_free=298.000000M/300M [09:58:52] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [10:13:42] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:15:31] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49924 MB (5% inode=99%): [10:17:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:19:43] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:13:51] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:15:32] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49789 MB (5% inode=99%): [11:17:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:19:53] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:52:03] [[Special:Log/newusers]] create 10 * Bishnu Saiki * (New user account) [11:52:44] nosy: could you pls update [[Admin:Next maintenance]] if anything has been done so far? thx [11:53:23] i particularly wonder about the zeus->apache switch [12:13:53] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:15:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49671 MB (5% inode=99%): [12:18:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:20:53] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:21:52] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:23] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:33] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:43] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:43] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:22:52] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:22:53] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:23:13] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 3522 MB (99% inode=99%): [12:23:23] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:23:23] SMTP on z-dat-s3-a is OK: SMTP OK - 0.003 sec. response time [12:23:32] SMTP on hyacinth is OK: SMTP OK - 0.002 sec. response time [12:23:32] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:23:42] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:23:43] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [13:13:53] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:15:33] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50476 MB (5% inode=99%): [13:18:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:20:53] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:54:02] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.060547/1.00, alarm hl:np_load_long=0.601562/1.50, alarm hl:mem_free=23669.000000M/300M [13:55:03] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [14:14:53] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:15:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50328 MB (5% inode=99%): [14:19:13] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:20:53] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:13:03] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=0.437500/1.75, alarm hl:np_load_avg=0.390137/2.00, alarm hl:mem_free=259.000000M/300M [15:14:03] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [15:15:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:16:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50178 MB (5% inode=99%): [15:20:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:21:03] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:15:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:17:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50024 MB (5% inode=99%): [16:20:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:22:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:47:12] Sun Grid Engine execd on nightshade is WARNING: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=1.570312/1.50, alarm hl:np_load_long=0.880859/1.75, alarm hl:mem_free=1526.000000M/250M [16:49:12] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [17:15:13] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:17:43] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49871 MB (5% inode=99%): [17:20:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:21:34] Load avg. on adenia is WARNING: WARNING - load average: 18.23, 11.61, 6.66 [17:22:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:23:34] Load avg. on adenia is OK: OK - load average: 14.00, 12.32, 7.55 [17:42:24] [[Special:Log/newusers]] create 10 * Chaipau * (New user account) [18:15:13] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:17:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49725 MB (5% inode=99%): [18:20:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:22:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:16:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:17:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49583 MB (5% inode=99%): [19:20:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:22:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:11:32] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 28323 MB (2% inode=99%): [20:16:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:18:52] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50363 MB (5% inode=99%): [20:20:32] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:23:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:17:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:18:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50198 MB (5% inode=99%): [21:20:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:23:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:18:12] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:18:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50063 MB (5% inode=99%): [22:20:33] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:23:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:27:23] 3(commented) [DBQ-173] problem with fr.wiki data base <10https://jira.toolserver.org/browse/DBQ-173> (madman) [22:29:22] 3(commented) [DBQ-175] checking cellistbot's it.wiki edits <10https://jira.toolserver.org/browse/DBQ-175> (madman) [22:38:44] zzz =_= [23:18:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:19:09] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49915 MB (5% inode=99%): [23:20:37] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:23:26] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default