[00:03:21] 3(created) [TS-1265] primary and secondary s1 database slaves aren't replication anymore; Toolserver: Databases; Blocker Bug <10https://jira.toolserver.org/browse/TS-1265> (merl) [00:06:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:08:20] 3(commented) [TS-1265] primary and secondary s1 database slaves aren't replication anymore <10https://jira.toolserver.org/browse/TS-1265> (merl) [00:09:03] Hmm. [00:09:14] Merlissimo: You sure? [00:09:51] Replag will be off all day because of no recent edits. Did they do a master switch today while the activity was low? [00:10:52] Joan: yes, you can read at tech log. And there are still edits on enwiki during the hole day. [00:11:01] @replag [00:11:01] Joan: s1-pri: 3m 26s [+0.18 s/s]; s1-sec: 3m 26s [+0.18 s/s]; s3-rr: 1m 29s [+0.15 s/s]; s3-user: 1m 29s [+0.15 s/s]; s5-rr: 17s [-0.00 s/s]; s5-user: 17s [-0.00 s/s]; s6-rr: 19s [+0.00 s/s]; s6-user: 19s [+0.00 s/s] [00:11:43] Joan: many user right changes on enwiki today [00:12:24] Heh, I saw. [00:12:30] Silliness. [00:12:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55898 MB (5% inode=99%): [00:33:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [00:54:23] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=15.114746/1.75, alarm hl:np_load_avg=14.940918/2.00, alarm hl:mem_free=1984.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=15.114746/1.50, alarm hl:np_load_long=14.801758/1.75, alarm hl:mem_free=1984.000000M/250M [00:54:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:56:23] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 120.52, 121.07, 119.25 [00:56:52] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:06:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:13:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55879 MB (5% inode=99%): [01:18:22] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1960.000000 [01:18:42] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1979.000000 [01:33:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:46:20] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3640.000000 [01:46:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3660.000000 [01:55:22] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=14.658691/1.75, alarm hl:np_load_avg=14.772461/2.00, alarm hl:mem_free=2288.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=14.658691/1.50, alarm hl:np_load_long=14.810059/1.75, alarm hl:mem_free=2288.000000M/250M [01:55:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:57:21] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 121.46, 119.75, 119.07 [01:57:50] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:06:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:10:23] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 22.000000 [02:10:41] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 41.000000 [02:13:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55848 MB (5% inode=99%): [02:33:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [02:55:01] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:01] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:22] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=14.585938/1.75, alarm hl:np_load_avg=14.766602/2.00, alarm hl:mem_free=2054.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=14.585938/1.50, alarm hl:np_load_long=14.671875/1.75, alarm hl:mem_free=2054.000000M/250M [02:55:31] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:55:52] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:55:52] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [02:55:52] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:56:01] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [02:57:23] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 110.39, 115.81, 116.64 [02:57:51] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:06:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:12:30] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:12:51] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:02] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:02] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:02] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:03] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:30] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:13:30] /v/sql on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:13:30] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:13:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55826 MB (5% inode=99%): [03:14:10] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [03:14:22] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [03:14:22] /v/sql on hyacinth is OK: DISK OK - free space: /v/sql 214239 MB (22% inode=99%): [03:14:22] /tmp on hyacinth is OK: DISK OK - free space: /tmp 3494 MB (99% inode=99%): [03:14:22] SMTP on z-dat-s3-a is OK: SMTP OK - 0.002 sec. response time [03:14:22] MySQL on z-dat-s3-a is OK: Uptime: 3328492 Threads: 9 Questions: 3584877960 Slow queries: 298416 Opens: 45275878 Flush tables: 2 Open tables: 16384 Queries per second avg: 1077.27 [03:14:30] MySQL slave on z-dat-s3-a is OK: Uptime: 3328496 Threads: 8 Questions: 3584879555 Slow queries: 298418 Opens: 45275881 Flush tables: 2 Open tables: 16384 Queries per second avg: 1077.26 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 254 [03:14:42] SMTP on z-dat-s7-a is OK: SMTP OK - 0.002 sec. response time [03:14:51] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:14:51] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:28:01] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:01] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:28:01] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:01] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:10] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:28:31] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [03:28:41] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 3630 MB (99% inode=99%): [03:28:51] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [03:56:20] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=15.052734/1.75, alarm hl:np_load_avg=15.158691/2.00, alarm hl:mem_free=2144.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=15.052734/1.50, alarm hl:np_load_long=17.232422/1.75, alarm hl:mem_free=2144.000000M/250M [03:56:50] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:58:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:58:20] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 114.73, 118.37, 134.43 [04:03:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:06:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:11:20] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1946.000000 [04:11:42] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1965.000000 [04:13:20] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 16.000000 [04:13:41] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 35.000000 [04:14:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55807 MB (5% inode=99%): [04:56:20] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_short=13.457520/1.75, alarm hl:np_load_avg=14.146973/2.00, alarm hl:mem_free=2194.000000M/300M: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=13.457520/1.50, alarm hl:np_load_long=14.637695/1.75, alarm hl:mem_free=2194.000000M/250M [04:57:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:58:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:58:20] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 114.68, 113.29, 116.55 [05:03:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:02] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:15:40] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55716 MB (5% inode=99%): [05:56:20] Sun Grid Engine execd on nightshade is WARNING: longrun@nightshade exceedes load threshold: alarm hl:np_load_short=0.719727/1.50, alarm hl:np_load_long=3.494629/1.75, alarm hl:mem_free=2407.000000M/250M [05:58:02] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:58:02] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:58:22] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 6.52, 6.66, 25.11 [06:04:22] Load avg. on nightshade is WARNING: WARNING - load average: 7.15, 7.73, 19.66 [06:06:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:11:20] Load avg. on nightshade is OK: OK - load average: 6.53, 6.72, 14.70 [06:13:20] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [06:15:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55571 MB (5% inode=99%): [06:18:00] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_short=1.846191/1.75, alarm hl:np_load_avg=1.646484/2.00, alarm hl:mem_free=1716.000000M/300M: longrun@willow exceedes load threshold: alarm hl:np_load_short=1.846191/1.50, alarm hl:np_load_long=1.349609/1.75, alarm hl:mem_free=1716.000000M/250M [06:20:01] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [06:33:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:59:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:59:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:07:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:15:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55445 MB (5% inode=99%): [07:33:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:42:35] [[Toolserver talk:Homepage]] ! 10https://wiki.toolserver.org/w/index.php?diff=6552&oldid=3427&rcid=8618 * 138.217.83.123 * (+3) () [07:42:51] [[Toolserver talk:Homepage]] ! 10https://wiki.toolserver.org/w/index.php?diff=6553&oldid=6552&rcid=8619 * 138.217.83.123 * (-3) () [07:59:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:59:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:07:50] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 41973 MB (4% inode=99%): [08:07:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:16:41] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55236 MB (5% inode=99%): [08:59:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:59:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:03:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:08:01] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:13:57] where are the dumps? [09:14:06] sftp://matanya@nightshade.toolserver.org/mnt/user-store/dump? [09:14:12] I see no hewiki there [09:17:40] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 55048 MB (5% inode=99%): [09:37:37] any TS admins here? [09:37:47] jepp [09:38:50] matanya: whats up? [09:38:59] hi nosy [09:39:21] I need the hewiki latest dump, but can't find it in the dump dir [09:43:38] nosy: ^ ? [09:46:30] matanya: http://dumps.wikimedia.org/ [09:46:51] I know that, the q is can I add it to the store [09:47:52] matanya: yes somewhere in /mnt/user-store/dumps/ [09:48:02] thank you [09:59:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:59:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:03:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:08:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:17:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54064 MB (5% inode=99%): [10:21:01] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:01] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:01] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:51] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:21:51] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [10:21:51] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [11:00:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:00:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:03:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:08:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:18:50] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53176 MB (5% inode=99%): [11:56:23] 3(created) [TS-1266] Quota for la2; Toolserver; Minor Task <10https://jira.toolserver.org/browse/TS-1266> (LA2) [12:01:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:01:00] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:03:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:08:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:18:01] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:01] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:30] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:41] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:18:50] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53337 MB (5% inode=99%): [12:19:00] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:19:00] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:19:00] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:19:10] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [12:19:10] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [12:19:21] MySQL on z-dat-s3-a is OK: Uptime: 3361185 Threads: 40 Questions: 3609123089 Slow queries: 300996 Opens: 45632423 Flush tables: 2 Open tables: 16384 Queries per second avg: 1073.765 [12:19:23] MySQL slave on z-dat-s3-a is OK: Uptime: 3361185 Threads: 40 Questions: 3609123089 Slow queries: 300996 Opens: 45632423 Flush tables: 2 Open tables: 16384 Queries per second avg: 1073.765 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 292 [12:19:23] SMTP on z-dat-s4-a is OK: SMTP OK - 0.070 sec. response time [12:19:29] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:19:41] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [12:28:00] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:28:01] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:01:01] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:02:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:03:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:07:30] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:08:21] SMTP on z-dat-s4-a is OK: SMTP OK - 0.003 sec. response time [13:18:50] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 52623 MB (5% inode=99%): [14:01:00] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:02:01] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:08:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:18:51] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51891 MB (5% inode=99%): [14:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:01:10] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:02:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:08:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:19:01] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 51109 MB (5% inode=99%): [15:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:01:13] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:02:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:08:11] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:20:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 50079 MB (5% inode=99%): [16:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:47:40] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:22] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:22] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:22] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:48:31] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:41] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:00] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [16:49:11] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:49:11] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [16:49:20] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:49:20] SMTP on hyacinth is OK: SMTP OK - 0.008 sec. response time [16:49:30] SMTP on z-dat-s4-a is OK: SMTP OK - 0.108 sec. response time [16:59:55] [[OpenStreetMap/Priorities 2012]] ! 10https://wiki.toolserver.org/w/index.php?diff=6554&oldid=6542&rcid=8620 * 84.50.62.182 * (+25) (/* Multilingual Maps */ ) [17:02:11] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:02:12] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:05:40] Sun Grid Engine execd on ortelius is WARNING: short@ortelius exceedes load threshold: alarm hl:np_load_short=1.296875/1.00, alarm hl:np_load_long=0.623047/1.50, alarm hl:mem_free=14624.000000M/300M: all.q@ortelius exceedes load threshold: alarm hl:np_load_short=1.296875/1.10, alarm hl:np_load_long=0.623047/1.75, alarm hl:mem_free=14624.000000M/300M [17:06:40] Sun Grid Engine execd on ortelius is OK: short@ortelius OK: all.q@ortelius OK [17:09:10] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:12:40] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:13:11] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:21] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:21] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:13:21] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:31] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:14:00] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [17:14:00] SMTP on z-dat-s7-a is OK: SMTP OK - 0.002 sec. response time [17:14:11] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [17:14:11] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [17:19:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49643 MB (5% inode=99%): [17:33:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:03:11] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:03:12] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:09:22] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:20:01] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 48871 MB (5% inode=99%): [18:33:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:50:26] If I run a query that looks like "select count(*), ll_from from langlinks group by 2 order by 1 desc limit 20;" [18:50:58] But I don't want the limit but I do want count(*) to be greater than 10, say, [18:51:05] what's the correct syntax for that? [18:51:53] [[Scs-oe10]] 10https://wiki.toolserver.org/w/index.php?diff=6555&oldid=6297&rcid=8621 * DaB * (+95) (Updated and corrected) [19:00:21] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/8:DOWN: 1 int NOK : CRITICAL [19:00:50] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/16:DOWN: 1 int NOK : CRITICAL [19:03:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:03:21] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:08:20] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/8:UP:1 UP: OK [19:08:51] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/16:UP:1 UP: OK [19:10:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:17:20] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/8:DOWN: 1 int NOK : CRITICAL [19:17:50] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/16:DOWN: 1 int NOK : CRITICAL [19:20:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 49067 MB (5% inode=99%): [20:03:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:04:21] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:04:22] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:04:28] Hello [20:04:38] We will shutdown willow in 1 minute [20:08:00] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 40929 MB (4% inode=99%): [20:08:51] / on willow is CRITICAL: Connection refused by host [20:08:51] SMTP on willow is CRITICAL: Connection refused [20:09:00] /tmp on willow is CRITICAL: Connection refused by host [20:09:15] Load avg. on willow is CRITICAL: Connection refused by host [20:09:15] SSH on willow is CRITICAL: Connection refused [20:09:21] Sun Grid Engine execd on willow is CRITICAL: Connection refused by host [20:09:21] SMF on willow is CRITICAL: Connection refused by host [20:09:41] FMA on willow is CRITICAL: ERROR - unexpected output from snmpwalk [20:09:41] NTP on willow is CRITICAL: NTP CRITICAL: No response from NTP server [20:10:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:13:51] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/16:DOWN: 1 int NOK : CRITICAL [20:15:31] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/1:DOWN: 1 int NOK : CRITICAL [20:15:41] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:DOWN: 1 int NOK : CRITICAL [20:17:21] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/8:DOWN: 1 int NOK : CRITICAL [20:20:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 47876 MB (4% inode=99%): [20:22:51] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/16:UP:1 UP: OK [20:23:20] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/8:UP:1 UP: OK [20:28:20] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/8:DOWN: 1 int NOK : CRITICAL [20:28:51] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/16:DOWN: 1 int NOK : CRITICAL [20:32:31] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/1:UP:1 UP: OK [20:32:40] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/14:UP:1 UP: OK [20:34:56] [[User:Faris knight]] ! 10https://wiki.toolserver.org/w/index.php?diff=6556&oldid=4859&rcid=8622 * Faris knight * (+159) () [20:35:31] Load avg. on willow is OK: OK - load average: 2.85, 0.74, 0.26 [20:35:31] FMA on willow is OK: OK [20:35:41] NTP on willow is OK: NTP OK: Offset -7e-06 secs [20:35:51] / on willow is OK: DISK OK - free space: / 35537 MB (32% inode=99%): [20:35:51] SMTP on willow is OK: SMTP OK - 0.133 sec. response time [20:36:00] /tmp on willow is OK: DISK OK - free space: /tmp 511 MB (99% inode=99%): [20:36:11] SSH on willow is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [20:40:31] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/1:DOWN: 1 int NOK : CRITICAL [20:40:41] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is CRITICAL: GigabitEthernet0/1/14:DOWN: 1 int NOK : CRITICAL [20:41:31] ethernet 0/1/1 [willow] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/1:UP:1 UP: OK [20:41:41] ethernet 0/1/14 [willow.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/14:UP:1 UP: OK [20:42:20] ethernet 0/1/8 [yarrow] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/8:UP:1 UP: OK [20:42:51] ethernet 0/1/16 [yarrow.mgmt] on asw-oe10-esams.mgmt is OK: GigabitEthernet0/1/16:UP:1 UP: OK [20:46:20] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [20:46:20] SMF on willow is OK: OK - all services online [20:48:13] zzz =_= [20:53:02] [[Scs-oe10]] 10https://wiki.toolserver.org/w/index.php?diff=6557&oldid=6555&rcid=8623 * DaB * (+23) (26 ist nun belegt) [21:03:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [21:04:23] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:04:23] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:10:21] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:21:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46600 MB (4% inode=99%): [21:54:20] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:56:10] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [21:58:25] [[Scs-oe10]] 10https://wiki.toolserver.org/w/index.php?diff=6558&oldid=6557&rcid=8624 * DaB * (+24) (ts-array3 temporär nicht erreichbar, Kabel wurde für SAN-switch gebraucht) [22:02:30] [[Scs-oe10]] 10https://wiki.toolserver.org/w/index.php?diff=6559&oldid=6558&rcid=8625 * DaB * (+324) (|scs-oe10 vs. |scs-oe16) [22:05:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:05:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:10:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:21:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 46685 MB (4% inode=99%): [22:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:05:20] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:05:20] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:10:20] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:21:00] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 45938 MB (4% inode=99%): [23:33:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:43:20] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:43:21] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:22] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:22] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:22] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:22] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:29] Environment on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:43:42] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:43:42] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:43:42] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:43:42] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:43:42] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:44:01] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [23:44:10] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:21] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:21] Environment on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [23:44:30] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [23:44:30] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [23:44:30] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [23:45:10] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [23:45:12] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [23:45:53] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [23:45:53] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [23:45:53] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:45:53] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:46:00] SMTP on z-dat-s6-a is OK: SMTP OK - 1.537 sec. response time [23:46:30] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 3490 MB (99% inode=99%): [23:46:30] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 3489 MB (99% inode=99%): [23:46:30] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 3488 MB (99% inode=99%): [23:46:30] MySQL on z-dat-s4-a is OK: Uptime: 4586558 Threads: 11 Questions: 195013131 Slow queries: 49667 Opens: 35173 Flush tables: 1 Open tables: 442 Queries per second avg: 42.518 [23:46:30] MySQL slave on z-dat-s4-a is OK: Uptime: 4586558 Threads: 11 Questions: 195013130 Slow queries: 49667 Opens: 35173 Flush tables: 1 Open tables: 442 Queries per second avg: 42.518 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 259 [23:46:30] MySQL slave on z-dat-s3-a is OK: Uptime: 3402423 Threads: 19 Questions: 3666531479 Slow queries: 304244 Opens: 46510205 Flush tables: 2 Open tables: 16384 Queries per second avg: 1077.623 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 368 [23:46:30] MySQL on z-dat-s3-a is OK: Uptime: 3402423 Threads: 19 Questions: 3666531481 Slow queries: 304244 Opens: 46510205 Flush tables: 2 Open tables: 16384 Queries per second avg: 1077.623 [23:46:31] MySQL slave on z-dat-s7-a is OK: Uptime: 2174857 Threads: 14 Questions: 502635360 Slow queries: 87972 Opens: 5942619 Flush tables: 1 Open tables: 3837 Queries per second avg: 231.111 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 387 [23:46:41] SMTP on z-dat-s4-a is OK: SMTP OK - 0.331 sec. response time [23:46:41] SMTP on z-dat-s3-a is OK: SMTP OK - 0.346 sec. response time [23:46:52] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [23:47:00] MySQL on z-dat-s6-a is OK: Uptime: 4669364 Threads: 11 Questions: 882824722 Slow queries: 311270 Opens: 8906236 Flush tables: 2 Open tables: 2894 Queries per second avg: 189.67 [23:47:01] MySQL slave on z-dat-s6-a is OK: Uptime: 4669364 Threads: 11 Questions: 882824723 Slow queries: 311270 Opens: 8906236 Flush tables: 2 Open tables: 2894 Queries per second avg: 189.67 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 351 [23:47:01] MySQL on z-dat-s7-a is OK: Uptime: 2174882 Threads: 18 Questions: 502637497 Slow queries: 87985 Opens: 5942710 Flush tables: 1 Open tables: 3837 Queries per second avg: 231.110 [23:47:10] SMTP on z-dat-s7-a is OK: SMTP OK - 0.004 sec. response time [23:47:11] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 126945 MB (31% inode=99%): [23:47:11] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 3594 MB (99% inode=99%): [23:47:11] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [23:47:11] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [23:47:11] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [23:47:11] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [23:47:12] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [23:55:20] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:22] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:55:22] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:22] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:22] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:22] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:22] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:42] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:55:42] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:55:52] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:41] /tmp on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:41] /tmp on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:41] /tmp on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:56:52] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [23:56:52] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [23:56:52] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [23:57:19] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [23:57:20] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [23:57:21] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out)