[00:02:59] Sun Grid Engine execd on willow is WARNING: all.q@willow exceedes load threshold: alarm hl:np_load_avg=1.543457/1.50, alarm hl:mem_free=635.000000M/100M [00:03:58] Sun Grid Engine execd on willow is OK: all.q@willow OK: longrun@willow OK [00:25:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:26:47] SSH on nightshade.mgmt is CRITICAL: Server answer: [00:30:39] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:34:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [00:35:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60739 MB (6% inode=99%): [01:25:47] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:26:59] SSH on nightshade.mgmt is CRITICAL: Server answer: [01:30:48] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:34:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:35:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60572 MB (6% inode=99%): [02:25:48] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:27:07] SSH on nightshade.mgmt is CRITICAL: Server answer: [02:30:48] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:36:17] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60411 MB (6% inode=99%): [03:04:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [03:25:57] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:27:07] SSH on nightshade.mgmt is CRITICAL: Server answer: [03:31:48] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:36:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60229 MB (6% inode=99%): [04:25:58] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:27:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [04:31:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:34:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [04:36:19] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60098 MB (6% inode=99%): [05:26:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:27:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [05:32:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:34:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:36:18] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59811 MB (6% inode=99%): [06:03:17] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.674316/1.50, alarm hl:mem_free=1378.000000M/100M [06:11:27] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [06:26:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:27:17] SSH on nightshade.mgmt is CRITICAL: Server answer: [06:30:26] Sun Grid Engine execd on nightshade is WARNING: all.q@nightshade exceedes load threshold: alarm hl:np_load_avg=1.646973/1.50, alarm hl:mem_free=1242.000000M/100M [06:32:26] Sun Grid Engine execd on nightshade is OK: all.q@nightshade OK: longrun@nightshade OK [06:32:58] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:34:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [06:37:17] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59849 MB (6% inode=99%): [07:26:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:26:17] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 60650 MB (6% inode=99%): [07:27:16] SSH on nightshade.mgmt is CRITICAL: Server answer: [07:34:08] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:34:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [07:37:28] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60650 MB (6% inode=99%): [08:27:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:27:27] SSH on nightshade.mgmt is CRITICAL: Server answer: [08:34:57] fisheye.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 274 bytes in 18.226 second response time [08:35:07] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:37:28] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60543 MB (6% inode=99%): [08:37:46] fisheye.toolserver.org on web.amaranth is OK: HTTP OK: HTTP/1.1 200 OK - 273 bytes in 10.424 second response time [09:04:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [09:21:37] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.420899/1.00, alarm hl:mem_free=12556.000000M/100M [09:27:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:27:37] SSH on nightshade.mgmt is CRITICAL: Server answer: [09:30:38] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [09:35:07] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:37:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60418 MB (6% inode=99%): [10:04:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [10:27:16] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:27:37] SSH on nightshade.mgmt is CRITICAL: Server answer: [10:35:07] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:38:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60299 MB (6% inode=99%): [11:04:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [11:27:17] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:27:37] SSH on nightshade.mgmt is CRITICAL: Server answer: [11:35:18] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:38:39] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60166 MB (6% inode=99%): [12:04:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [12:23:49] Sun Grid Engine execd on ortelius is WARNING: all.q@ortelius exceedes load threshold: alarm hl:np_load_avg=1.148438/1.00, alarm hl:mem_free=11460.000000M/100M [12:27:27] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:27:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [12:29:49] Sun Grid Engine execd on ortelius is OK: all.q@ortelius OK [12:35:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:39:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60004 MB (6% inode=99%): [13:04:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [13:27:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:27:49] SSH on nightshade.mgmt is CRITICAL: Server answer: [13:35:28] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:39:37] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59853 MB (6% inode=99%): [14:18:20] 3(resolved) [MAGNUS-272] File Upload Bot (Magnus Manske) sets wrong rotate template when importing flickr <10https://jira.toolserver.org/browse/MAGNUS-272> (Magnus Manske) [14:27:28] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:28:48] SSH on nightshade.mgmt is CRITICAL: Server answer: [14:34:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [14:35:31] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:39:38] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59703 MB (6% inode=99%): [15:27:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:29:00] SSH on nightshade.mgmt is CRITICAL: Server answer: [15:34:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [15:35:30] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:39:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60339 MB (6% inode=99%): [15:40:55] DaBPunkt: i get a "query interrupted" error even though my query contains /* SLOW_OK */ on sql-s1-rr. any idea what could be wrong? [15:42:31] Daniel_WMDE: "SLOW_OK" doesn't means that the query will never interrupted, it depens on the replag of the host [15:42:57] yea, i just fopund that in the documentation. hm... [15:43:06] but i should be getting an email, right? [15:43:10] yes [15:43:14] even without LIMIT. [15:43:52] @replag [15:43:53] Daniel_WMDE: s1-sec: 9m 40s [+0.00 s/s]; s1-sec-c: 47s [-]; s3-rr: 3m 17s [+0.00 s/s]; s3-user: 3m 17s [+0.00 s/s]; s4-rr: 47s [-] [15:44:27] hm... [15:47:04] DaBPunkt: lag on sql-s1-rr seems to be rather high. why doesn't @replag report it? what's s1-sec anyway? [15:47:36] Daniel_WMDE: s1-sec is rosemary [15:47:44] which is also sql-s1-rr at the moment [15:48:31] nop [15:48:32] e [15:48:45] you like to bet? ;) [15:48:45] DaBPunkt: try mysql -h sql-s1-rr -e "select @@global.hostname" [15:48:59] err, wait [15:49:19] you are right :) [15:49:28] but sql-s1-user is thyme [15:49:41] yes, and? [15:50:39] well, one of them is missing in @replag, right? [15:51:09] Daniel_WMDE: yes and no. If there is no replag, @replag does not show the host at all. [15:51:17] @report all [15:51:45] ah, ok. and rr is lagged 9 minutes, so it's shown [15:51:53] Daniel_WMDE: yes [15:51:58] ok, thanks :) [15:52:17] normaly "@report all" should show ALL replag, no idea why it doe not at the moment… [15:53:56] (the replag of rosemary is decreasing BTW, so the chance is high that your query will finish this time) [15:54:30] i switched to -user, makes more sens for that app anyway [15:55:01] hm... i hope i'm not *causing* the replag [15:55:15] i'm running some slow queries, causing a filesort [15:55:28] mm, -user is normaly more loaded so the replag is normaly higher too [15:55:33] shouldn't block the table from updating though [15:55:56] there is still a query by you on rosemary BTW [15:56:46] I installed your software BTW yesterday [16:09:52] DaBPunkt: thanks [16:12:32] the query worked this time, btw [16:13:05] very slow though. [16:20:59] Daniel_WMDE: guppy/heapy funzt nicht :( [16:21:02] siehe jira [16:22:03] but it's not urgent. it's just a profiling tool, i don't need it right now. [16:22:11] but it would be nice to have it around [16:22:20] 3(commented) [TS-1255] install guppy for python <10https://jira.toolserver.org/browse/TS-1255> (Daniel Kinzler) [16:22:21] 3(reopened) [TS-1255] install guppy for python <10https://jira.toolserver.org/browse/TS-1255> (Daniel Kinzler) [16:22:29] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1942.000000 [16:23:05] great… [16:27:38] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:29:58] SSH on nightshade.mgmt is CRITICAL: Server answer: [16:32:29] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000 [16:34:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [16:35:40] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:35:45] Daniel_WMDE: looks like a 64-bit-problem. There are bug-reports in redhat, debian and in the guppy-tracker. There is also a patch in thier svn. I will patch the programm [16:36:25] 3(work started) [TS-1255] install guppy for python <10https://jira.toolserver.org/browse/TS-1255> (DaB.) [16:36:27] 3(commented) [TS-1255] install guppy for python <10https://jira.toolserver.org/browse/TS-1255> (Daniel Kinzler) [16:36:41] DaBPunkt: thanks [16:39:48] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60316 MB (6% inode=99%): [16:52:48] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:53:18] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [16:55:24] 3(resolved) [TS-1255] install guppy for python <10https://jira.toolserver.org/browse/TS-1255> (DaB.) [16:57:03] toolserver: 03dab * r1127 10/trunk/ts-specs/ (base-specs/python-guppy.spec patches/guppy-0.diff): [16:57:03] toolserver: - Added patch for guppy that let it run also on 64-bit-archs. Taken from rev. [16:57:03] toolserver: 82 of the guppy-svn, author svenil. [17:10:28] 3(commented) [TS-1255] install guppy for python <10https://jira.toolserver.org/browse/TS-1255> (DaB.) [17:24:00] danke DaBPunkt [17:24:17] np [17:25:21] perhaps put a link to the patch into the jira ticket, for later reference [17:27:48] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:29:57] SSH on nightshade.mgmt is CRITICAL: Server answer: [17:34:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [17:35:48] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:39:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60141 MB (6% inode=99%): [18:28:48] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:30:08] SSH on nightshade.mgmt is CRITICAL: Server answer: [18:33:04] any geohack dev around? [18:34:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:35:48] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:36:28] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1928.000000 [18:36:28] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1928.000000 [18:39:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60028 MB (6% inode=99%): [18:56:48] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:57:27] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:04:38] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [19:04:39] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [19:07:09] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [19:26:58] /aux0 on daphne is CRITICAL: DISK CRITICAL - free space: /aux0 61456 MB (6% inode=99%): [19:28:48] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:30:08] SSH on nightshade.mgmt is CRITICAL: Server answer: [19:34:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [19:35:49] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:39:58] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59820 MB (6% inode=99%): [20:04:50] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7220.000000 [20:04:50] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7220.000000 [20:08:08] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [20:28:58] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:30:07] SSH on nightshade.mgmt is CRITICAL: Server answer: [20:34:09] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [20:35:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:39:59] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 59672 MB (6% inode=99%): [21:05:48] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10881.000000 [21:05:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10886.000000 [21:08:17] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [21:20:21] 3(created) [MNT-1171] Switch master for s4; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1171> (DaB.) [21:21:52] z-dat-s4-a changed [21:21:58] NTP on turnera.esi is WARNING: NTP WARNING: Server not synchronized, Offset 0.001033 secs [21:29:59] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:30:25] rosemary is changed [21:30:57] NTP on turnera.esi is OK: NTP OK: Offset 0.000742 secs [21:30:59] [[Admin:Change the replication master]] 10https://wiki.toolserver.org/w/index.php?diff=6520&oldid=6185&rcid=8570 * Nosy * (+0) (/* If the master host changed */ ) [21:31:07] SSH on nightshade.mgmt is CRITICAL: Server answer: [21:33:55] daphne is changed [21:33:58] @replag [21:33:59] DaBPunkt: s1-pri: 24s [-]; s1-sec-c: 2h 42m 57s [+0.46 s/s]; s2/s5-pri-c: 2h 52m 45s [+0.01 s/s]; s3-rr: 17s [-0.01 s/s]; s3-user: 17s [-0.01 s/s]; s4-rr: 2h 42m 57s [+0.46 s/s]; s4-user: 2h 4m 37s [-] [21:34:21] @replag [21:34:22] DaBPunkt: s1-pri: 21s [-0.13 s/s]; s1-sec-c: 2h 42m 39s [-0.79 s/s]; s2/s5-pri-c: 2h 52m 48s [+0.13 s/s]; s3-rr: 40s [+1.00 s/s]; s3-user: 40s [+1.00 s/s]; s4-rr: 2h 42m 39s [-0.79 s/s]; s4-user: 2h 2m 14s [-6.24 s/s] [21:36:59] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:37:21] 3(updated) [MNT-1171] Switch master for s4 <10https://jira.toolserver.org/browse/MNT-1171> (DaB.) [21:40:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60398 MB (6% inode=99%): [21:51:18] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3375 [21:54:24] @replag [21:54:24] DaBPunkt: s1-sec-c: 1h 55m 21s [-2.36 s/s]; s2/s5-pri-c: 2h 51m 13s [-0.08 s/s]; s3-rr: 20s [-0.02 s/s]; s3-user: 20s [-0.02 s/s]; s4-rr: 1h 55m 21s [-2.36 s/s]; s4-user: 43m 18s [-3.94 s/s] [21:58:18] MySQL slave on z-dat-s4-a is OK: Uptime: 3456859 Threads: 5 Questions: 144485457 Slow queries: 37807 Opens: 25741 Flush tables: 1 Open tables: 438 Queries per second avg: 41.796 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1541 [22:04:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [22:05:48] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4611.000000 [22:05:48] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9737.000000 [22:13:49] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3596.000000 [22:16:35] @replag [22:16:35] DaBPunkt: s1-sec: 18s [-0.02 s/s]; s1-sec-c: 54m 31s [-2.74 s/s]; s2/s5-pri-c: 2h 23m 6s [-1.27 s/s]; s3-rr: 48s [+0.02 s/s]; s3-user: 48s [+0.02 s/s]; s4-rr: 54m 31s [-2.74 s/s] [22:24:49] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1606.000000 [22:30:07] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:31:07] SSH on nightshade.mgmt is CRITICAL: Server answer: [22:37:07] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:40:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60179 MB (6% inode=99%): [22:48:48] @replag [22:48:49] DaBPunkt: s2/s5-pri-c: 1h 16m 37s [-2.06 s/s]; s3-rr: 34s [-0.01 s/s]; s3-user: 34s [-0.01 s/s] [22:58:49] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3504.000000 [23:04:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [23:09:57] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1682.000000 [23:30:19] SMF on turnera.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:31:19] SSH on nightshade.mgmt is CRITICAL: Server answer: [23:37:19] SMF on damiana.esi is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:40:07] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 60040 MB (6% inode=99%):