[00:01:25] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.91, 22.07, 19.51 [00:02:26] Load avg. on willow is WARNING: WARNING - load average: 24.71, 22.05, 19.68 [00:09:15] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 183640 MB (3% inode=23%): [00:30:56] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [00:36:05] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [00:37:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [00:42:35] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [00:43:25] /tmp on willow is WARNING: DISK WARNING - free space: / 20606 MB (19% inode=98%): [00:44:37] / on willow is WARNING: DISK WARNING - free space: / 20604 MB (19% inode=98%): [00:48:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:02:56] Free Memory on turnera is WARNING: WARNING - 6.8% (283560 kB) free! [01:03:24] Load avg. on willow is WARNING: WARNING - load average: 19.25, 19.61, 18.94 [01:03:55] Free Memory on turnera is OK: OK - 7.1% (296484 kB) free. [01:09:30] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 183010 MB (3% inode=23%): [01:10:28] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.92, 21.53, 20.09 [01:28:28] Load avg. on willow is WARNING: WARNING - load average: 18.49, 19.61, 19.95 [01:30:29] Load avg. on willow is CRITICAL: CRITICAL - load average: 30.16, 21.81, 20.63 [01:31:07] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [01:36:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [01:38:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [01:42:47] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [01:43:38] /tmp on willow is WARNING: DISK WARNING - free space: / 20492 MB (19% inode=98%): [01:44:49] / on willow is WARNING: DISK WARNING - free space: / 20489 MB (19% inode=98%): [01:48:37] Load avg. on willow is WARNING: WARNING - load average: 18.33, 19.23, 19.91 [01:48:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:50:38] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.35, 20.47, 20.25 [01:59:37] Load avg. on willow is WARNING: WARNING - load average: 17.84, 19.58, 20.00 [02:09:38] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 180795 MB (3% inode=23%): [02:19:51] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.65, 22.16, 21.87 [02:31:08] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [02:36:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [02:38:18] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [02:43:16] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [02:43:58] /tmp on willow is WARNING: DISK WARNING - free space: / 20388 MB (19% inode=98%): [02:44:59] Load avg. on willow is WARNING: WARNING - load average: 16.54, 18.36, 19.98 [02:44:59] / on willow is WARNING: DISK WARNING - free space: / 20387 MB (19% inode=98%): [02:45:59] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.60, 19.88, 20.42 [02:49:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:06:54] [[Special:Log/newusers]] create 10 * Aarpdental * (New user account) [03:09:57] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 176998 MB (3% inode=22%): [03:31:27] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [03:36:27] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [03:38:38] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [03:42:38] Free Memory on turnera is WARNING: WARNING - 6.0% (249304 kB) free! [03:43:37] Free Memory on turnera is OK: OK - 7.3% (305944 kB) free. [03:43:58] /tmp on willow is WARNING: DISK WARNING - free space: / 20283 MB (19% inode=98%): [03:44:06] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [03:45:06] / on willow is WARNING: DISK WARNING - free space: / 20280 MB (19% inode=98%): [03:46:56] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.92, 24.38, 24.52 [03:50:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:53:37] Free Memory on turnera is WARNING: WARNING - 6.6% (276224 kB) free! [04:05:38] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.346680/1.10, alarm hl:np_load_long=0.878906/1.55, alarm hl:mem_free=8871.000000M/500M, alarm hl:tmp_free=15572M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.346680/1.00, alarm hl:np_load_long=0.878906/1.50, alarm hl:mem_free=8871.000000M/600M, alarm hl:tmp_free=1 [04:06:47] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [04:09:57] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 176719 MB (3% inode=22%): [04:16:35] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44689 MB (10% inode=99%): [04:18:35] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 73153 MB (18% inode=99%): [04:31:47] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [04:33:47] Free Memory on turnera is WARNING: WARNING - 6.1% (254072 kB) free! [04:35:47] Free Memory on turnera is OK: OK - 8.4% (353300 kB) free. [04:36:48] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [04:39:01] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [04:44:09] /tmp on willow is WARNING: DISK WARNING - free space: / 19905 MB (18% inode=98%): [04:44:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [04:45:11] / on willow is WARNING: DISK WARNING - free space: / 19897 MB (18% inode=98%): [04:47:10] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.08, 24.59, 24.37 [04:50:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:55:16] [[User:Aarpdental]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7346&rcid=9856 * Aarpdental * (+5014) (Created page with "Outstanding news with regards to Humana Dental Insurance protection I feel I have an understanding of you would like dental insurance policy. Immediately following all, can be...") [05:04:58] [[Special:Log/newusers]] create 10 * Samuele Papa * (New user account) [05:10:09] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 175740 MB (3% inode=22%): [05:31:49] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [05:36:49] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [05:39:09] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [05:44:09] /tmp on willow is WARNING: DISK WARNING - free space: / 19641 MB (18% inode=98%): [05:44:40] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [05:46:09] / on willow is WARNING: DISK WARNING - free space: / 19638 MB (18% inode=98%): [05:48:09] Load avg. on willow is CRITICAL: CRITICAL - load average: 20.68, 20.04, 20.40 [05:50:09] Load avg. on willow is WARNING: WARNING - load average: 16.92, 18.88, 19.93 [05:51:10] Load avg. on willow is CRITICAL: CRITICAL - load average: 18.90, 19.32, 20.03 [05:51:39] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:52:09] Load avg. on willow is WARNING: WARNING - load average: 18.17, 18.98, 19.86 [05:57:49] /sql on cassia is WARNING: DISK WARNING - free space: /sql 112382 MB (9% inode=99%): [06:03:50] Free Memory on turnera is WARNING: WARNING - 5.6% (235488 kB) free! [06:05:09] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.88, 20.83, 20.03 [06:07:49] Free Memory on turnera is OK: OK - 13.5% (564432 kB) free. [06:10:09] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 175965 MB (3% inode=22%): [06:13:59] Free Memory on turnera is WARNING: WARNING - 6.2% (259812 kB) free! [06:28:09] Load avg. on willow is WARNING: WARNING - load average: 16.04, 18.04, 19.73 [06:31:09] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.78, 20.25, 20.21 [06:31:59] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [06:36:20] Load avg. on willow is WARNING: WARNING - load average: 18.36, 19.45, 19.89 [06:37:02] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [06:39:20] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [06:44:10] /tmp on willow is WARNING: DISK WARNING - free space: / 19404 MB (18% inode=98%): [06:45:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [06:46:21] / on willow is WARNING: DISK WARNING - free space: / 19399 MB (18% inode=98%): [06:51:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:10:19] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 176105 MB (3% inode=22%): [07:13:19] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.30, 21.09, 20.03 [07:14:19] Load avg. on willow is WARNING: WARNING - load average: 19.49, 20.52, 19.89 [07:15:20] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.96, 21.36, 20.20 [07:24:49] Free Memory on turnera is WARNING: WARNING - 6.2% (261076 kB) free! [07:26:51] Free Memory on turnera is OK: OK - 7.1% (298812 kB) free. [07:31:49] Free Memory on turnera is WARNING: WARNING - 6.5% (273052 kB) free! [07:32:11] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [07:32:40] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:37:59] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [07:39:30] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [07:44:29] /tmp on willow is WARNING: DISK WARNING - free space: / 19297 MB (18% inode=98%): [07:46:30] / on willow is WARNING: DISK WARNING - free space: / 19293 MB (18% inode=98%): [07:46:39] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [07:47:09] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [07:47:29] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.02, 23.78, 22.44 [07:52:40] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:10:30] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 175537 MB (3% inode=22%): [08:19:30] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1851.000000 [08:33:09] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [08:38:10] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [08:39:46] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [08:44:11] Free Memory on turnera is WARNING: WARNING - 6.2% (260380 kB) free! [08:44:30] /tmp on willow is WARNING: DISK WARNING - free space: / 19201 MB (18% inode=98%): [08:45:09] Free Memory on turnera is OK: OK - 7.6% (316696 kB) free. [08:46:30] / on willow is WARNING: DISK WARNING - free space: / 19198 MB (18% inode=98%): [08:46:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [08:47:31] Load avg. on willow is CRITICAL: CRITICAL - load average: 28.26, 27.81, 25.12 [08:52:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:03:09] Free Memory on turnera is WARNING: WARNING - 6.0% (252360 kB) free! [09:11:29] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 175640 MB (3% inode=22%): [09:19:39] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2608.000000 [09:33:19] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [09:38:20] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [09:39:50] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [09:44:39] /tmp on willow is WARNING: DISK WARNING - free space: / 19105 MB (18% inode=98%): [09:46:39] / on willow is WARNING: DISK WARNING - free space: / 19102 MB (18% inode=98%): [09:46:59] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [09:47:39] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.02, 21.58, 23.34 [09:52:50] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:50] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:51] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:51] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:52:51] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:52:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:53:10] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:10] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:20] Environment IPMI on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:20] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:29] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:29] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:29] / on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:40] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [09:53:40] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [09:53:40] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:40] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:40] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:50] SMF on z-dat-s4-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:50] SMF on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:50] /tmp on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:50] SMF on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:50] SMF on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:53:51] Load avg. on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:54:29] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [09:54:30] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [09:54:40] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [09:54:50] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [09:54:50] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [09:55:20] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:29] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:39] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [09:55:40] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [09:57:59] /tmp on z-dat-s7-a is OK: DISK OK - free space: /tmp 924 MB (99% inode=99%): [09:57:59] / on hyacinth is OK: DISK OK - free space: / 8270 MB (27% inode=85%): [09:58:09] MySQL on z-dat-s3-a is OK: Uptime: 2900240 Threads: 24 Questions: 3075521016 Slow queries: 194409 Opens: 30035940 Flush tables: 1 Open tables: 16384 Queries per second avg: 1060.436 [09:58:09] MySQL slave on z-dat-s3-a is OK: Uptime: 2900240 Threads: 24 Questions: 3075521016 Slow queries: 194409 Opens: 30035940 Flush tables: 1 Open tables: 16384 Queries per second avg: 1060.436 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1388 [09:58:09] MySQL on z-dat-s4-a is OK: Uptime: 561796 Threads: 12 Questions: 19280969 Slow queries: 5785 Opens: 15892 Flush tables: 1 Open tables: 790 Queries per second avg: 34.320 [09:58:09] MySQL slave on z-dat-s4-a is OK: Uptime: 561796 Threads: 12 Questions: 19280969 Slow queries: 5785 Opens: 15892 Flush tables: 1 Open tables: 790 Queries per second avg: 34.320 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 409 [09:58:09] SMTP on z-dat-s7-a is OK: SMTP OK - 0.036 sec. response time [09:58:09] / on z-dat-s7-a is OK: DISK OK - free space: / 8270 MB (27% inode=85%): [09:58:09] SMF on z-dat-s7-a is OK: OK - all services online [09:58:10] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:58:20] s4 replag on z-dat-s4-a is OK: QUERY OK: SELECT ts_rc_age() returned 399.000000 [09:58:21] /tmp on hyacinth is OK: DISK OK - free space: /tmp 983 MB (100% inode=99%): [09:58:21] SMF on z-dat-s4-a is OK: OK - all services online [09:58:21] SMF on z-dat-s3-a is OK: OK - all services online [09:58:21] SMF on z-dat-s6-a is OK: OK - all services online [09:58:21] SMF on hyacinth is OK: OK - all services online [09:58:21] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [09:58:21] SMTP on hyacinth is OK: SMTP OK - 0.149 sec. response time [09:58:22] SMTP on z-dat-s4-a is OK: SMTP OK - 0.005 sec. response time [09:58:22] Load avg. on hyacinth is OK: OK - load average: 0.87, 0.64, 1.19 [09:58:29] MySQL on z-dat-s7-a is OK: Uptime: 540184 Threads: 17 Questions: 161285414 Slow queries: 21403 Opens: 1233875 Flush tables: 1 Open tables: 3995 Queries per second avg: 298.574 [09:58:29] MySQL on z-dat-s6-a is OK: Uptime: 959762 Threads: 15 Questions: 193539314 Slow queries: 66402 Opens: 1794357 Flush tables: 1 Open tables: 2517 Queries per second avg: 201.653 [09:58:29] MySQL slave on z-dat-s7-a is OK: Uptime: 540184 Threads: 18 Questions: 161285414 Slow queries: 21403 Opens: 1233875 Flush tables: 1 Open tables: 3995 Queries per second avg: 298.574 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1379 [09:58:29] MySQL slave on z-dat-s6-a is OK: Uptime: 959762 Threads: 15 Questions: 193539315 Slow queries: 66402 Opens: 1794357 Flush tables: 1 Open tables: 2517 Queries per second avg: 201.653 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 533 [09:58:29] SMTP on z-dat-s3-a is OK: SMTP OK - 0.070 sec. response time [09:58:40] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:58:40] Load avg. on z-dat-s7-a is OK: OK - load average: 1.36, 0.75, 1.22 [09:58:40] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:58:40] /sql on z-dat-s7-a is OK: DISK OK - free space: /sql 85475 MB (21% inode=99%): [09:58:40] SSH on z-dat-s7-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:58:41] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [09:58:59] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [10:05:50] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1784.000000 [10:07:23] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:32] /tmp on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:07:32] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:07:41] / on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:07:41] SMF on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:07:42] RAID on hyacinth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:08:12] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:14] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:14] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:14] NTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:23] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [10:08:24] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [10:08:32] Load avg. on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:08:32] /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:08:32] s4 replag on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [10:08:42] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [10:08:42] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [10:08:42] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [10:08:42] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [10:08:42] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:08:42] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:03] NTP on hyacinth is OK: NTP OK: Offset -0.000654 secs [10:09:32] SMTP on z-dat-s6-a is OK: SMTP OK - 0.043 sec. response time [10:11:41] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 175377 MB (3% inode=22%): [10:27:51] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2057 [10:33:32] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [10:35:52] MySQL slave on z-dat-s7-a is OK: Uptime: 542425 Threads: 11 Questions: 161886237 Slow queries: 21588 Opens: 1233909 Flush tables: 1 Open tables: 3995 Queries per second avg: 298.449 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1772 [10:38:33] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [10:40:12] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [10:45:03] /tmp on willow is WARNING: DISK WARNING - free space: / 18991 MB (18% inode=98%): [10:47:02] / on willow is WARNING: DISK WARNING - free space: / 18988 MB (18% inode=98%): [10:47:02] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [10:48:04] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.21, 24.27, 23.39 [10:53:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:05:33] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.437500/1.10, alarm hl:np_load_long=0.658203/1.55, alarm hl:mem_free=8785.000000M/500M, alarm hl:tmp_free=15379M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=1.437500/1.00, alarm hl:np_load_long=0.658203/1.50, alarm hl:mem_free=8785.000000M/600M, alarm hl:tmp_free=1 [11:07:33] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [11:11:43] 3(commented) [ACCAPP-514] Perform database queries across English and Telugu wiki projects for identifying quality improvement <10https://jira.toolserver.org/browse/ACCAPP-514> (Arjuna Rao Chavala) [11:12:02] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 199049 MB (3% inode=24%): [11:21:48] [[Special:Log/newusers]] create 10 * Abdallae773 * (New user account) [11:22:44] 3(updated) [ACCAPP-514] Perform database queries across English and Telugu wiki projects for identifying quality improvement <10https://jira.toolserver.org/browse/ACCAPP-514> (Arjuna Rao Chavala) [11:34:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [11:38:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [11:40:23] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [11:45:02] /tmp on willow is WARNING: DISK WARNING - free space: / 18868 MB (18% inode=98%): [11:47:03] / on willow is WARNING: DISK WARNING - free space: / 18865 MB (18% inode=98%): [11:47:12] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [11:49:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.15, 26.16, 27.81 [11:53:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:53:46] interwiki bots are banned, but I still see a lot of them zzz [11:53:54] this is not fairplay [12:00:43] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:01:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:10:13] Sun Grid Engine execd on wolfsbane is WARNING: medium-sol@wolfsbane exceedes load threshold: alarm hl:np_load_short=0.319824/1.00, alarm hl:np_load_long=0.246582/1.50, alarm hl:mem_free=540.000000M/600M, alarm hl:tmp_free=15077M/100M, alarm hl:available=1/0 [12:11:05] hello all [12:12:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 198840 MB (3% inode=24%): [12:12:13] Sun Grid Engine execd on wolfsbane is OK: short-sol@wolfsbane OK: medium-sol@wolfsbane OK [12:12:43] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:15:00] hello DaB [12:20:02] Load avg. on willow is WARNING: WARNING - load average: 13.77, 16.96, 19.91 [12:21:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.00, 18.61, 20.32 [12:23:02] Load avg. on willow is WARNING: WARNING - load average: 18.62, 18.13, 19.91 [12:34:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [12:37:33] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:38:42] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [12:40:32] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [12:45:02] /tmp on willow is WARNING: DISK WARNING - free space: / 18774 MB (17% inode=98%): [12:47:11] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [12:48:02] / on willow is WARNING: DISK WARNING - free space: / 18769 MB (17% inode=98%): [12:56:02] / on willow is OK: DISK OK - free space: / 22744 MB (21% inode=99%): [12:56:02] /tmp on willow is OK: DISK OK - free space: / 22744 MB (21% inode=99%): [13:00:02] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1861.000000 [13:02:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:13:03] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 198635 MB (3% inode=24%): [13:21:22] Free Memory on turnera is WARNING: WARNING - 6.1% (254680 kB) free! [13:23:02] Load avg. on willow is WARNING: WARNING - load average: 17.44, 19.22, 19.16 [13:26:22] Free Memory on turnera is OK: OK - 10.1% (423800 kB) free. [13:34:52] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [13:38:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [13:40:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [13:47:15] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [13:54:23] Free Memory on turnera is WARNING: WARNING - 6.9% (287780 kB) free! [13:56:24] Free Memory on turnera is OK: OK - 7.3% (304664 kB) free. [14:00:15] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3494.000000 [14:02:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:04:24] Free Memory on turnera is WARNING: WARNING - 6.6% (276984 kB) free! [14:13:14] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 198393 MB (3% inode=24%): [14:14:14] Load avg. on willow is OK: OK - load average: 12.54, 13.48, 14.90 [14:19:14] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3638.000000 [14:33:15] Load avg. on willow is WARNING: WARNING - load average: 19.39, 17.48, 15.42 [14:35:15] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [14:37:14] Load avg. on willow is OK: OK - load average: 12.18, 14.64, 14.68 [14:38:04] Sun Grid Engine execd on ortelius is WARNING: short-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.092773/1.10, alarm hl:np_load_long=0.878906/1.55, alarm hl:mem_free=9118.000000M/500M, alarm hl:tmp_free=15290M/200M, alarm hl:available=1/0: medium-sol@ortelius exceedes load threshold: alarm hl:np_load_short=2.092773/1.00, alarm hl:np_load_long=0.878906/1.50, alarm hl:mem_free=9118.000000M/600M, alarm hl:tmp_free=1 [14:39:03] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [14:41:03] Sun Grid Engine execd on ortelius is OK: short-sol@ortelius OK: medium-sol@ortelius OK [14:41:45] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [14:47:24] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [14:58:48] [[Special:Log/newusers]] create 10 * Base * (New user account) [15:00:35] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [15:01:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:02:14] Load avg. on willow is WARNING: WARNING - load average: 18.35, 14.70, 13.61 [15:04:14] Load avg. on willow is OK: OK - load average: 14.81, 14.51, 13.67 [15:13:14] Load avg. on willow is WARNING: WARNING - load average: 15.48, 14.84, 14.08 [15:13:14] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 198022 MB (3% inode=24%): [15:19:15] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6724.000000 [15:35:16] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [15:39:04] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [15:41:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [15:47:25] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [15:53:45] Free Memory on turnera is WARNING: WARNING - 6.9% (288112 kB) free! [15:55:44] Free Memory on turnera is OK: OK - 7.2% (302932 kB) free. [15:56:34] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 43562 MB (10% inode=99%): [16:00:34] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 74300 MB (18% inode=99%): [16:01:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:03:24] Load avg. on willow is WARNING: WARNING - load average: 15.31, 14.93, 14.42 [16:03:44] Free Memory on turnera is WARNING: WARNING - 6.2% (259936 kB) free! [16:04:25] Load avg. on willow is OK: OK - load average: 14.50, 14.70, 14.37 [16:13:26] Load avg. on willow is WARNING: WARNING - load average: 17.91, 15.68, 14.88 [16:13:26] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 197918 MB (3% inode=24%): [16:13:47] [[Special:Log/newusers]] create 10 * Styroks * (New user account) [16:19:25] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9648.000000 [16:26:02] @replag [16:26:02] Dispenser: s1-user: 12s [+0.00 s/s]; s2-user: 4d 16h 52m 28s [+0.74 s/s]; s2-user-c: 2h 46m 16s [+0.11 s/s]; s3-rr-a: 1m 19s [+0.00 s/s]; s3-user: 1m 19s [+0.00 s/s]; s5-user-c: 2h 46m 16s [+0.11 s/s] [16:35:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [16:39:13] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [16:42:04] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [16:43:25] Load avg. on willow is WARNING: WARNING - load average: 15.25, 14.96, 14.66 [16:44:24] Load avg. on willow is OK: OK - load average: 14.18, 14.73, 14.60 [16:44:46] Free Memory on turnera is WARNING: WARNING - 6.8% (282536 kB) free! [16:46:45] Free Memory on turnera is OK: OK - 7.1% (296976 kB) free. [16:47:24] Load avg. on willow is WARNING: WARNING - load average: 15.79, 15.55, 14.97 [16:47:34] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:01:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:14:25] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 197742 MB (3% inode=24%): [17:18:24] Load avg. on willow is WARNING: WARNING - load average: 13.12, 14.71, 15.16 [17:20:24] Load avg. on willow is OK: OK - load average: 14.02, 14.45, 15.00 [17:20:25] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12435.000000 [17:22:46] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:23:24] Load avg. on willow is WARNING: WARNING - load average: 15.72, 15.33, 15.29 [17:27:24] Load avg. on willow is OK: OK - load average: 12.87, 14.32, 14.89 [17:36:24] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [17:37:16] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [17:39:14] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [17:42:15] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [17:43:04] Free Memory on turnera is WARNING: WARNING - 6.0% (251784 kB) free! [17:45:24] [[Special:Log/newusers]] create 10 * Arubavacation * (New user account) [17:47:04] Free Memory on turnera is OK: OK - 7.3% (304000 kB) free. [17:47:44] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [17:49:56] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User:Aarpdental10]]": spam) [17:50:10] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Aarpdental10]] with an expiry time of infinite (account creation disabled): inappropriate behavior) [17:58:25] Load avg. on willow is WARNING: WARNING - load average: 13.42, 15.11, 15.18 [17:58:34] /sql on cassia is WARNING: DISK WARNING - free space: /sql 113677 MB (9% inode=99%): [18:02:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:05:05] Free Memory on turnera is WARNING: WARNING - 6.9% (288912 kB) free! [18:06:05] Free Memory on turnera is OK: OK - 7.1% (297932 kB) free. [18:15:25] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 197569 MB (3% inode=24%): [18:20:25] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15316.000000 [18:26:04] Free Memory on turnera is WARNING: WARNING - 6.6% (275184 kB) free! [18:27:05] Free Memory on turnera is OK: OK - 7.7% (321416 kB) free. [18:36:44] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [18:39:26] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [18:42:44] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [18:44:24] Free Memory on turnera is WARNING: WARNING - 6.7% (280032 kB) free! [18:47:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [18:55:44] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 39749 MB (9% inode=99%): [18:58:26] Load avg. on willow is WARNING: WARNING - load average: 15.17, 15.62, 16.23 [19:02:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:09:54] /sql on z-dat-s4-a is CRITICAL: DISK CRITICAL - free space: /sql 23733 MB (5% inode=99%): [19:10:53] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:11:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:13:19] [[User:Arubavacation]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7347&rcid=9864 * Arubavacation * (+3171) (Created page with "An important productive sector that dominates daily life inside the Caribbean is tourism. Lots of islands rely on earnings generated by the move of visitors they be given from t...") [19:15:34] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 197381 MB (3% inode=24%): [19:15:52] [[Special:Log/delete]] delete 10 * Gifti * (deleted "[[02User:Arubavacation10]]": Spam) [19:16:19] Grr. [19:17:43] heh [19:20:34] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17714.000000 [19:23:53] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 90474 MB (22% inode=99%): [19:26:14] Joan: I know several ways to reduce the amount of spam on wiki [19:27:11] Joan: https://jira.toolserver.org/browse/TS-1392 [19:27:50] Betacommand: Seems rather stupid. [19:27:57] I saw your edits. They were bad hacks. [19:28:12] If you want to limit page creation to autoconfirmed users or whatever, use LocalSettings.php, not some Titleblacklist hackery. [19:28:35] Joan: at the time I thought the TBL was enabled [19:30:19] Right. [19:30:44] 3(commented) [TS-1392] Install Extension:TitleBlacklist <10https://jira.toolserver.org/browse/TS-1392> (MZMcBride) [19:32:54] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:37:35] Load avg. on willow is OK: OK - load average: 12.76, 14.15, 14.92 [19:37:43] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [19:40:25] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [19:42:54] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [19:47:24] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [19:47:35] Load avg. on willow is WARNING: WARNING - load average: 16.59, 15.01, 14.81 [19:47:54] SMF on damiana is CRITICAL: ERROR - maintenance: svc:/network/ldap/client:default offline: svc:/system/cluster/scsymon-srv:default [19:48:34] Load avg. on willow is OK: OK - load average: 14.16, 14.64, 14.69 [20:11:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:15:34] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 197107 MB (3% inode=24%): [20:15:55] /sql on z-dat-s4-a is WARNING: DISK WARNING - free space: /sql 44395 MB (10% inode=99%): [20:19:42] 3(created) [MNT-1247] Enabled ldap-client on damiana; Maintenance; Emergency work <10https://jira.toolserver.org/browse/MNT-1247> (DaB.) [20:19:55] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 72861 MB (18% inode=99%): [20:20:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20024.000000 [20:37:53] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:40:43] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [20:43:11] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [20:45:43] Free Memory on turnera is WARNING: WARNING - 6.4% (268176 kB) free! [20:48:02] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [20:48:44] Free Memory on turnera is OK: OK - 7.2% (302956 kB) free. [20:53:43] Free Memory on turnera is WARNING: WARNING - 5.8% (241780 kB) free! [21:08:47] @replag [21:08:48] DaBPunkt: s1-user: 14s [+0.00 s/s]; s2-user: 4d 19h 54m 50s [+0.64 s/s]; s2-user-c: 5h 25m 58s [+0.56 s/s]; s3-rr-a: 12s [-0.00 s/s]; s3-user: 12s [-0.00 s/s]; s5-user-c: 5h 25m 58s [+0.56 s/s] [21:11:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:15:53] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 196941 MB (3% inode=24%): [21:20:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19511.000000 [21:38:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:41:08] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [21:44:08] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [21:49:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [21:55:08] SMF on cassia is CRITICAL: ERROR - maintenance: svc:/network/trainwreck:s2 [22:01:08] SMF on cassia is OK: OK - all services online [22:05:08] Free Memory on turnera is WARNING: WARNING - 6.6% (277684 kB) free! [22:06:08] Free Memory on turnera is OK: OK - 7.7% (321892 kB) free. [22:12:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:13:07] @replag [22:13:07] DaBPunkt: s1-user: 6m 27s [+0.34 s/s]; s2-user: 4d 20h 16m 2s [+0.43 s/s]; s2-user-c: 4h 52m 50s [-0.45 s/s]; s3-rr-a: 12s [-0.16 s/s]; s3-user: 12s [-0.16 s/s]; s5-user-c: 4h 52m 50s [-0.45 s/s] [22:16:17] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 196764 MB (3% inode=24%): [22:21:00] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17632.000000 [22:26:40] nacht ts [22:27:03] nacht DaBPunkt [22:38:01] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:41:18] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [22:44:28] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [22:49:58] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [22:54:18] Free Memory on turnera is WARNING: WARNING - 6.1% (256848 kB) free! [22:57:18] Free Memory on turnera is OK: OK - 7.1% (297492 kB) free. [23:02:18] Load avg. on willow is WARNING: WARNING - load average: 16.88, 15.29, 13.22 [23:03:18] Load avg. on willow is OK: OK - load average: 14.60, 14.88, 13.20 [23:04:18] Free Memory on turnera is WARNING: WARNING - 6.1% (256504 kB) free! [23:12:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:16:18] /aux0 on hemlock is CRITICAL: DISK CRITICAL - free space: /aux0 196166 MB (3% inode=24%): [23:21:59] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11977.000000 [23:39:00] SMF on turnera is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default [23:41:39] SMF on willow is CRITICAL: ERROR - maintenance: svc:/network/puppetmasterd:default [23:44:48] FMA on yarrow is CRITICAL: ERROR - unexpected output from snmpwalk [23:51:00] SMF on damiana is CRITICAL: ERROR - offline: svc:/system/cluster/scsymon-srv:default