[00:00:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:10:59] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:13:09] Free Memory on damiana is WARNING: WARNING - 5.4% (455056 kB) free! [00:15:09] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (408708 kB) free! [00:16:10] Free Memory on damiana is WARNING: WARNING - 5.6% (472428 kB) free! [00:16:49] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:20:09] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:20:10] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:20:10] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:20:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:20:10] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:20:39] aliasd on yarrow is CRITICAL: Connection refused [00:21:40] SSH on mayapple is CRITICAL: Server answer: [00:21:49] MySQL on rosemary is OK: Uptime: 1609 Threads: 3 Questions: 37 Slow queries: 0 Opens: 18 Flush tables: 1 Open tables: 11 Queries per second avg: 0.22 [00:21:49] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2301 [00:21:59] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2302.000000 [00:22:09] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (401736 kB) free! [00:23:40] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [00:26:49] MySQL slave on rosemary is OK: Uptime: 1900 Threads: 18 Questions: 67969 Slow queries: 91 Opens: 291 Flush tables: 1 Open tables: 257 Queries per second avg: 35.773 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1777 [00:26:50] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1776.000000 [00:26:50] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:33:40] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [00:34:40] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [00:36:09] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:36:19] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:36:29] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [00:37:49] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:37:50] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:37:50] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:38:19] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:38:20] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:38:20] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:38:40] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:39:19] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:39:19] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:39:19] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:39:40] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:39:40] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:39:40] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:40] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:41] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:41] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:43:09] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [00:45:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:46:19] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:46:59] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:47:20] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:47:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:47:50] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:47:50] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:19] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:19] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:40] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:48:40] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:48:41] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:10] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:10] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:20] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:49:40] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 865360.000000 [00:49:40] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:49:40] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:49:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:50:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1340578.000000 [00:50:39] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 633792.000000 [00:50:40] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:50:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:50:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:51:20] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:40] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:51:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:52:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 797141.000000 [00:52:09] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:52:19] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:52:30] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 391797.000000 [00:52:40] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:52:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:53:10] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:53:10] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:53:10] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:53:19] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2039886.000000 [00:53:40] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:53:40] / on thyme is UNKNOWN: NRPE: Unable to read output [00:53:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:54:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:54:20] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:54:40] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:54:41] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52503 MB (8% inode=99%): [00:54:49] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:54:50] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:54:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:54:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:55:40] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1234064.000000 [00:55:40] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:40] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:20] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:56:41] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:56:41] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:56:41] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:57:19] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:57:39] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:57:40] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:57:41] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:58:20] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [00:58:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 265320 MB (4% inode=62%): [00:59:40] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [01:00:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:10:10] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:11:00] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:17:23] [[Special:Log/newusers]] create 10 * MrSoNord * (New user account) [01:19:10] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:20:10] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:20:11] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:20:11] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:20:12] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:20:12] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:21:05] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:21:29] FC 0/13 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:21:30] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:21:30] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:21:30] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:21:31] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:21:32] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:21:33] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:21:33] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:21:33] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:21:34] / on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:21:34] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:21:34] /tmp on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:21:34] APT on z-dat-s1-b is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:21:35] APT on yucca is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:21:35] APT on z-dat-s2-b is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:21:35] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:21:58] SSH on mayapple is CRITICAL: Server answer: [01:22:02] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:22:02] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [01:22:04] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [01:23:12] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [01:23:33] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:23:34] /sql on z-dat-s1-b is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:45:45] / on thyme is UNKNOWN: NRPE: Unable to read output [01:45:45] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:45] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:45] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:45] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:46] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:46] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [01:45:47] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:45:48] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:51] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:52] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:52] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:52] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:52] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [01:45:52] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:53] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:45:53] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:46:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:47:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:48:24] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:13] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:50:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:50:23] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 866569.000000 [01:50:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1343215.000000 [01:50:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:51:23] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 634651.000000 [01:51:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:51:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:51:54] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [01:52:14] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:52:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 800756.000000 [01:52:43] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 393148.000000 [01:52:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:53:29] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:53:44] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:53:44] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:54:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:54:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2042923.000000 [01:54:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [01:55:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52430 MB (8% inode=99%): [01:56:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1237185.000000 [01:56:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:58:32] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2175.000000 [01:58:59] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 265212 MB (4% inode=62%): [01:59:14] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2200.000000 [01:59:23] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2200.000000 [02:00:23] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2205.000000 [02:00:44] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [02:02:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:02:13] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [02:03:13] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:04:14] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2179.000000 [02:04:44] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:04:44] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:04:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:04:54] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:06:14] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1536.000000 [02:07:02] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2184.000000 [02:07:13] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:07:23] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1208.000000 [02:10:13] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:20:03] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:20:14] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:20:14] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:20:14] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:20:55] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:21:02] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:22:13] aliasd on yarrow is CRITICAL: Connection refused [02:22:23] SSH on mayapple is CRITICAL: Server answer: [02:22:33] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3615.000000 [02:22:43] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [02:24:23] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3645.000000 [02:24:33] /sql on z-dat-s1-b is CRITICAL: DISK CRITICAL - free space: /sql 57650 MB (5% inode=99%): [02:28:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3619.000000 [02:31:03] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3623.000000 [02:44:33] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2739.000000 [02:44:53] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:53] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:53] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:44:53] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:53] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:54] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:02] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:02] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:03] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:03] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:12] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:12] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:12] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:12] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:45:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:14] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:15] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:22] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:22] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:22] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:32] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:45:32] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 420.000000 [02:45:32] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:32] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:32] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:33] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:33] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:34] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:35] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:42] / on thyme is UNKNOWN: NRPE: Unable to read output [02:45:42] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:42] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:42] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [02:45:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:43] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:43] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:44] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:44] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:45:45] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:52] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:52] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:53] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:53] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:53] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [02:45:53] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:53] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:46:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:47:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:49:02] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3436.000000 [02:49:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:22] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3320.000000 [02:49:22] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:53] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4427 [02:50:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:50:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 866732.000000 [02:50:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1344261.000000 [02:50:32] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2864 [02:50:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:50:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4273 [02:50:52] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2877 [02:51:23] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 633168.000000 [02:51:43] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:51:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:51:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:52:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 804355.000000 [02:52:32] MySQL slave on daphne is OK: Uptime: 1494355 Threads: 60 Questions: 193771995 Slow queries: 340874 Opens: 63490 Flush tables: 1 Open tables: 1893 Queries per second avg: 129.669 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1718 [02:52:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 392990.000000 [02:52:52] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [02:53:02] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1260.000000 [02:53:13] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:53:48] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:54:02] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:54:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2045731.000000 [02:54:22] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1743.000000 [02:54:22] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:54:42] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:54:43] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:54:53] MySQL slave on z-dat-s5-b is OK: Uptime: 367263 Threads: 5 Questions: 621799788 Slow queries: 3756 Opens: 207376 Flush tables: 1 Open tables: 256 Queries per second avg: 1693.64 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1586 [02:55:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [02:56:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:56:12] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3536.000000 [02:56:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52380 MB (8% inode=99%): [02:56:43] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3463 [02:57:22] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1238215.000000 [02:57:22] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:59:52] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 265193 MB (4% inode=62%): [03:01:42] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:02:13] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:03:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [03:03:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:04:52] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3567 [03:05:43] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:05:43] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:05:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:05:53] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:07:43] MySQL slave on rosemary is OK: Uptime: 11554 Threads: 22 Questions: 6323284 Slow queries: 1688 Opens: 961 Flush tables: 1 Open tables: 502 Queries per second avg: 547.280 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1779 [03:08:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:08:12] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1714.000000 [03:10:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:13:12] aliasd on yarrow is OK: TCP OK - 0.007 second response time on port 984 [500 Not found.] [03:20:12] aliasd on yarrow is CRITICAL: Connection refused [03:21:03] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:21:03] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:21:12] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:21:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:21:13] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:21:52] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:22:22] SSH on mayapple is CRITICAL: Server answer: [03:23:13] aliasd on yarrow is OK: TCP OK - 0.007 second response time on port 984 [500 Not found.] [03:23:42] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [03:24:32] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 90317 MB (9% inode=99%): [03:34:02] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:39:53] MySQL slave on z-dat-s1-b is OK: Uptime: 370013 Threads: 24 Questions: 316428233 Slow queries: 7319 Opens: 1959 Flush tables: 1 Open tables: 448 Queries per second avg: 855.181 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1720 [03:44:52] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:44:53] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:44:53] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:44:53] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:44:53] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:44:53] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:12] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:12] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:12] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:45:13] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:13] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:14] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:14] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:15] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:22] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:22] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:32] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:45:33] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:33] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:33] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:33] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:33] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:33] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:34] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:34] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:43] / on thyme is UNKNOWN: NRPE: Unable to read output [03:45:43] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:43] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [03:45:43] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:43] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:43] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:44] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:44] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:45:45] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:53] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:53] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:53] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:53] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:53] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:45:53] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:54] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:03] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:46:03] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:46:03] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [03:46:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:03] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:47:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:49:12] aliasd on yarrow is CRITICAL: Connection refused [03:49:13] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:23] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:50:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:51:13] aliasd on yarrow is OK: TCP OK - 0.007 second response time on port 984 [500 Not found.] [03:51:23] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 631508.000000 [03:51:23] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 861813.000000 [03:51:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1344231.000000 [03:51:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:51:52] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:51:52] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:52:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 807955.000000 [03:52:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 392478.000000 [03:52:53] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [03:53:13] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:53:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:54:22] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:54:43] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:54:43] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:55:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:55:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2048724.000000 [03:55:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:55:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [03:56:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:56:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52261 MB (8% inode=99%): [03:58:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1239416.000000 [03:58:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:59:53] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 265094 MB (4% inode=62%): [04:01:43] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:02:12] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:03:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [04:03:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:05:42] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:05:42] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:05:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:05:53] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:08:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:10:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:21:03] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:21:03] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:21:12] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:21:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:21:13] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:21:52] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:22:22] SSH on mayapple is CRITICAL: Server answer: [04:23:42] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [04:35:02] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:44:53] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:44:53] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:44:53] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:44:53] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:53] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:54] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:12] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:12] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:12] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:45:12] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:45:13] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:14] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:14] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:15] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:22] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:22] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:42] / on thyme is UNKNOWN: NRPE: Unable to read output [04:45:43] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:43] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [04:45:43] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:43] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:43] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:44] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:45:44] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:45:45] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:53] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:53] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:53] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:53] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:53] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:45:53] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:03] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:03] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:03] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:03] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:32] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:46:32] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:32] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:32] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:33] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:33] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [04:46:34] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:34] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:47:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:49:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:23] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:51:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:51:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:51:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:52:22] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 629602.000000 [04:52:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 853946.000000 [04:52:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1344698.000000 [04:52:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 391775.000000 [04:52:52] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [04:53:13] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:53:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 811616.000000 [04:54:22] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:54:43] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:54:43] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:54:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:55:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:55:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2051533.000000 [04:55:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:56:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:56:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 52129 MB (8% inode=99%): [04:56:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [04:58:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:58:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240767.000000 [04:58:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:59:52] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 264997 MB (4% inode=62%): [05:01:42] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:02:13] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:03:03] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:03:15] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:05:43] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:05:43] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:05:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:05:53] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:08:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:10:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:15:32] /sql on z-dat-s1-b is CRITICAL: DISK CRITICAL - free space: /sql 58334 MB (5% inode=99%): [05:21:02] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:21:02] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:21:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:21:12] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:21:12] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:21:53] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:22:23] SSH on mayapple is CRITICAL: Server answer: [05:23:43] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [05:35:02] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:42] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:39:12] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [05:44:53] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:44:53] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:44:53] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:44:53] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:44:53] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:44:53] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:23] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:23] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:42] / on thyme is UNKNOWN: NRPE: Unable to read output [05:45:42] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:42] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:43] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [05:45:43] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:43] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:44] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:45:44] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:45:45] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:52] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:52] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:52] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:52] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:52] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:45:53] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:53] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:54] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:12] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:13] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:13] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:13] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:46:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:46:13] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:14] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:15] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:33] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:46:33] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:33] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:33] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:33] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:33] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:34] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:46:34] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:35] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:47:02] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:47:02] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:47:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:02] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [05:47:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:47:32] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 89840 MB (9% inode=99%): [05:47:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:49:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:49:22] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:50:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:51:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:51:43] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:51:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:52:22] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 626011.000000 [05:52:23] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 850844.000000 [05:52:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1345593.000000 [05:52:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 391621.000000 [05:52:52] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [05:53:12] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:53:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 815216.000000 [05:54:42] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:54:42] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:54:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:55:02] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:55:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2054168.000000 [05:55:22] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:56:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:56:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51960 MB (8% inode=99%): [05:57:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descr [05:58:22] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:58:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1241901.000000 [05:58:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:59:52] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 264503 MB (4% inode=62%): [06:01:42] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:03:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:04:13] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:05:43] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:05:43] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:05:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:05:53] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:08:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:10:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:21:02] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:21:02] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:21:12] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:21:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:21:12] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:21:53] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:22:22] SSH on mayapple is CRITICAL: Server answer: [06:23:43] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [06:27:33] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:28:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:36:02] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:12] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [06:44:52] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:44:52] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:44:53] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:44:53] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:44:53] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:44:53] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:23] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:23] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:42] / on thyme is UNKNOWN: NRPE: Unable to read output [06:45:43] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:43] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [06:45:43] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:43] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:44] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:44] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:45] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:45:45] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:53] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:53] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:53] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:53] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:53] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:45:54] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:54] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:55] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:12] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:12] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:12] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:46:13] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:13] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:14] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:14] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:15] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:32] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:46:32] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:32] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:32] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:33] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:33] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:33] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:46:34] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:34] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:46:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:47:03] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:47:03] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:47:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:03] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [06:47:03] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:49:13] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:49:22] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:51:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:51:52] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:51:52] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:52:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:52:22] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 620654.000000 [06:52:23] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 844362.000000 [06:52:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1346156.000000 [06:52:53] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [06:53:12] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:53:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 818816.000000 [06:53:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 390561.000000 [06:54:42] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:54:42] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:54:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:55:02] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:55:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2056379.000000 [06:55:22] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:56:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51627 MB (8% inode=99%): [06:57:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:57:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:59:22] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1243129.000000 [06:59:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:53] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 264357 MB (4% inode=62%): [07:01:43] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:04:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:05:42] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:05:42] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:05:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:05:53] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:08:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:10:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:21:03] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:21:03] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:21:13] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:21:13] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:21:13] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:21:53] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:22:22] SSH on mayapple is CRITICAL: Server answer: [07:23:42] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [07:28:03] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:31:03] Marlen Caemmerer * [Toolserver-l] Maintenance: Rebooting ortelius web server [07:32:32] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:33:03] Marlen Caemmerer * [Toolserver-l] Maintenance: Solaris Updates [07:33:12] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:34:32] APT on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:36:02] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:36:12] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:36:26] Load avg. on nightshade is WARNING: WARNING - load average: 25.01, 24.96, 16.43 [07:36:42] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [07:39:12] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:44:13] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 34.35, 21.39, 16.82 [07:44:52] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:44:52] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:44:53] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:44:53] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:44:53] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:44:53] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:12] Load avg. on nightshade is WARNING: WARNING - load average: 22.20, 20.17, 16.66 [07:45:22] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:22] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:43] / on thyme is UNKNOWN: NRPE: Unable to read output [07:45:43] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:43] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [07:45:43] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:43] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:44] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:44] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:45] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:45:45] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:53] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:53] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:53] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:53] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:53] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:45:53] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:45:54] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:13] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:13] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:13] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:13] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:46:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:46:14] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:14] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:15] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:32] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:46:32] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:32] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:33] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:33] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:33] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:46:34] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:34] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:46:53] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:02] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:47:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:02] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:47:02] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [07:47:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:12] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:12] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:49:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:49:22] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:32] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:51:43] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:51:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:51:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:52:22] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 840485.000000 [07:52:22] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 613580.000000 [07:52:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1346783.000000 [07:52:52] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [07:53:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:53:12] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:53:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 822417.000000 [07:53:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 389130.000000 [07:54:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [07:55:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:55:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2058903.000000 [07:55:42] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:55:42] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:55:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:56:22] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:57:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:57:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51507 MB (8% inode=99%): [07:57:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:58:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:22] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1243702.000000 [07:59:22] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:52] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 264098 MB (4% inode=62%): [08:01:42] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:04:13] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:05:42] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:05:43] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:05:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:05:53] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:08:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:10:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:21:02] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:21:03] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:21:12] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:21:12] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:21:12] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:21:22] Free Memory on turnera is WARNING: WARNING - 5.9% (497084 kB) free! [08:21:53] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:23:22] SSH on mayapple is CRITICAL: Server answer: [08:23:43] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [08:30:22] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (379492 kB) free! [08:32:22] Free Memory on turnera is WARNING: WARNING - 5.3% (444408 kB) free! [08:37:03] toolserver.org HTTP on wolfsbane is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:37:42] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:38:23] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:44:52] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:44:53] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:44:54] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:44:54] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:44:54] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:44:54] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:23] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:23] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:42] / on thyme is UNKNOWN: NRPE: Unable to read output [08:45:43] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:43] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:43] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [08:45:43] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:43] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:44] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:44] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:45:45] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:45:45] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:52] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:52] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:53] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:53] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:53] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:45:53] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:45:53] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:12] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:12] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:12] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:46:13] ethernet 0/1/14 [cassini.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:13] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:14] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:32] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:46:32] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:33] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:33] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:33] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:33] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:33] ethernet 0/1/17 [sage] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:46:34] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:34] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:53] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:13] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:13] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:48:02] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:48:02] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:02] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [08:48:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:49:12] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:49:22] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:50:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:51:42] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:52] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:51:52] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:52:23] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 837797.000000 [08:52:23] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 608727.000000 [08:52:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1346993.000000 [08:52:53] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [08:53:02] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:53:12] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:53:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 826017.000000 [08:53:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 386974.000000 [08:55:02] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [08:55:02] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:55:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2061284.000000 [08:55:42] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:43] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:55:52] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:57:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:22] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:57:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 51363 MB (8% inode=99%): [08:57:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:58:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:52] toolserver.org HTTP on wolfsbane is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.016 second response time [08:59:22] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:23] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1244456.000000 [08:59:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:53] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 263115 MB (4% inode=62%): [09:01:42] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [09:04:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:04:22] Free Memory on turnera is CRITICAL: CRITICAL - 3.9% (323984 kB) free! [09:04:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:05:42] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:05:42] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:05:52] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [09:08:23] Free Memory on turnera is WARNING: WARNING - 5.4% (451152 kB) free! [09:09:12] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:10:12] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:12:22] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (361112 kB) free! [11:05:10] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7991&oldid=7990&rcid=21874 * Beta16 * (+158) (/* …move somewhen this year. */ ) [12:24:26] [[Labs-Moving-Survey]] ! 10https://wiki.toolserver.org/w/index.php?diff=7992&oldid=7991&rcid=21875 * Mormegil * (+302) (/* …stay as long as possible. */ Bawolff++) [12:24:54] [[Labs-Moving-Survey]] !M 10https://wiki.toolserver.org/w/index.php?diff=7993&oldid=7992&rcid=21876 * Mormegil * (+3) (/* …stay as long as possible. */ oops, too many ~s) [13:41:46] [[Special:Log/newusers]] create 10 * Hypnotistes * (New user account) [13:43:05] [[User talk:Hypnotistes]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7994&rcid=21878 * Hypnotistes * (+77) (Created page with "[http://www.hypnotherapyhypnotist.com/ http://www.hypnotherapyhypnotist.com/]") [13:45:04] [[Special:Log/delete]] delete 10 * Betacommand * (deleted "[[02User talk:Hypnotistes10]]") [14:43:05] @notify DaBPunkt [14:43:05] I will notify you, when I see DaBPunkt around here [14:46:27] scfc_de: what do you need? [14:53:11] Betacommand: One of my scripts reported this morning "oursql.UnknownError: (126, "Incorrect key file for table '/sql/tmp/#sql_750d_1.MYI'; try to repair it", None)" [14:54:07] I don't have information on which host that occured, but it must have been one of enwiki_ps. [14:54:15] (Filing a bug in JIRA later.) [15:05:27] (For reference: https://jira.toolserver.org/browse/TS-1645) [15:21:19] hello all [15:23:45] DaBPunkt: Hi! Could you take a look at https://jira.toolserver.org/browse/TS-1645? [15:37:11] Merlissimo: Could you take a look at https://jira.toolserver.org/browse/TS-1642? I can't delete it either with (as operator) "sudo /.../qdel". [15:38:24] scfc_de: done [15:39:17] Merlissimo: And what's the magic word?! :-) [15:39:38] sge manager [15:40:17] Okay, thanks! [15:40:18] qdel -f ... [15:41:41] Could you please mention that on JIRA for future reference? [16:18:55] scfc_de: which table is this? [16:21:03] DaBPunkt: http://pastebin.com/8tP4F16j [16:22:09] great, that's a temporary table [16:24:10] So nothing broken? [16:27:47] AFAIS no. It is a disk-shortage-problem [16:29:08] ... that could affect the main tables? [16:31:34] I'm not sure [16:33:35] Getting 502 Bad Gateway nginx/1.0.4 on HTTPS for several GET and POST requests [16:35:03] And timeout over HTTP instead of HTTPS [16:36:20] example? [17:27:20] DaBPunkt: if I wanted to connect to s1-rr how would I do that via terminal? [17:45:07] Betacommand: "sql enwiki_p" should work. [17:47:23] scfc_de: the question is, is that s1-rr or s1-user [17:53:36] Betacommand: s1-rr is a superset of s1-user. If you just want to access s1-user, "sql -u enwiki_p". If you want to access a server with *no* user databases, I think you have to name the host explicitly. thyme seems to be down, rosemary has user databases, and z-dat-s1-a doesn't respond. [17:54:39] (Superset in the sense of "all hosts in s1-user are in s1-rr", not s1-rr has all data (= user databases) that s1-user has.) [17:56:23] Betacommand: Ah, z-dat-s1-b works and has *no* user databases. So use this if you need a host with no user databases. [18:00:09] scfc_de: the reason I was asking is s1-user's copy of enwiki is corrupt [18:01:05] Betacommand: Sorry, can't fix that :-). [18:36:52] "atq" on yarrow: "Cannot access user database: Transport endpoint is not connected" [18:37:25] willow asks for password [18:39:36] nightshade as well. [18:39:57] ... and the webserver returns 404s. [18:44:52] Still gone. [18:46:08] wait a moment [18:48:26] the ldap-service is away [19:03:13] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2700 [19:03:53] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2906.000000 [19:03:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1262468.000000 [19:04:02] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:11] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 369852.000000 [19:04:12] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [19:04:12] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:22] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2386 [19:04:32] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2281 [19:04:32] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2949.000000 [19:04:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:04:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49497 MB (8% inode=99%): [19:04:32] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2938.000000 [19:04:54] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 88423 MB (9% inode=99%): [19:04:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:04:54] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [19:04:54] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [19:04:54] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:04:54] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:04:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:04:54] aliasd on nightshade is CRITICAL: Connection refused [19:04:54] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [19:04:55] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [19:05:02] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:05:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:05:03] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:05:03] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:05:03] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:05:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 824379.000000 [19:05:22] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:05:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:32] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:05:32] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:05:32] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:05:32] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:05:32] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:05:33] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:05:33] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:05:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:05:41] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 601097.000000 [19:05:52] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:05:52] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:05:52] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:52] SSH on mayapple is CRITICAL: Server answer: [19:05:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:06:01] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:11] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:06:11] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:06:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:06:12] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:06:13] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:06:13] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:06:13] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:06:13] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:06:13] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:06:14] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:06:14] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:06:15] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:06:32] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [19:06:32] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:06:32] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:06:42] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:06:52] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:06:52] ... and we're back. [19:07:02] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:07:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:07:12] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:07:12] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:07:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:07:12] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:07:22] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:07:22] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:07:22] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:07:22] / on thyme is UNKNOWN: NRPE: Unable to read output [19:07:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:07:31] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:07:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:07:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:07:41] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:07:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:07:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:07:52] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:52] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [19:07:52] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:07:52] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:08:01] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:08:01] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:08:01] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:08:19] it should work again. Looks like one of the ha-nodes can not run the ldap-service anymore… [19:08:32] MySQL slave on z-dat-s3-a is OK: Uptime: 87288 Threads: 16 Questions: 134465485 Slow queries: 2807 Opens: 1080471 Flush tables: 1 Open tables: 16384 Queries per second avg: 1540.480 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1744 [19:09:19] It occurs from time to time. [19:11:31] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:12:22] MySQL slave on z-dat-s1-b is OK: Uptime: 425960 Threads: 9 Questions: 361567747 Slow queries: 8383 Opens: 2161 Flush tables: 1 Open tables: 472 Queries per second avg: 848.830 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1632 [19:15:31] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [19:15:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000 [19:16:31] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3658.000000 [19:21:12] MySQL slave on rosemary is OK: Uptime: 69970 Threads: 10 Questions: 64163508 Slow queries: 12127 Opens: 6603 Flush tables: 1 Open tables: 2881 Queries per second avg: 917.14 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1711 [19:21:41] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1661.000000 [19:23:31] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3228.000000 [19:23:52] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3584.000000 [19:24:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:24:32] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:24:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:26:33] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1187.000000 [19:28:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:29:55] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1748.000000 [19:55:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1363150.000000 [19:58:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 865944.000000 [20:00:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 259721 MB (4% inode=62%): [20:02:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2092481.000000 [20:04:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1265647.000000 [20:04:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 370403.000000 [20:04:12] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [20:04:12] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:12] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:04:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 49135 MB (8% inode=99%): [20:04:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:04:55] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [20:04:55] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:04:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [20:04:55] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:04:56] aliasd on nightshade is CRITICAL: Connection refused [20:04:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [20:04:57] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:05:03] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [20:05:03] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:05:03] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:05:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:05:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:05:17] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:05:17] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 823013.000000 [20:05:22] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:05:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:32] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:32] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:32] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:33] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:33] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:33] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:33] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:05:41] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 601692.000000 [20:05:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:05:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:05:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:55] SSH on mayapple is CRITICAL: Server answer: [20:05:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:06:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:13] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:06:13] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:06:13] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:06:13] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:06:13] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:06:13] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:06:14] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:06:14] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:06:15] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:06:15] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:06:15] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:06:33] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [20:06:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:06:33] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:06:42] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:06:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:07:03] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:07:13] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:07:13] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:07:13] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:07:13] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:07:13] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:07:22] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:07:22] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:07:22] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:07:22] / on thyme is UNKNOWN: NRPE: Unable to read output [20:07:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:07:32] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:07:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:07:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:07:41] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:07:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:07:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:07:55] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:55] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [20:07:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:07:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:08:03] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:08:03] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:08:03] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:16:32] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7258.000000 [20:24:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:24:32] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:24:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:28:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:55:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1363408.000000 [20:58:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 869543.000000 [21:00:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 259442 MB (4% inode=62%): [21:02:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2094340.000000 [21:04:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1268405.000000 [21:04:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 368628.000000 [21:04:12] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [21:04:13] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:04:13] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:04:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48914 MB (8% inode=99%): [21:04:33] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:04:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:04:55] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [21:04:55] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:04:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [21:04:55] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:04:56] aliasd on nightshade is CRITICAL: Connection refused [21:04:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [21:04:57] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:05:03] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [21:05:03] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:05:03] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:05:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:22] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:05:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 599737.000000 [21:05:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:05:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:05:55] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:55] SSH on mayapple is CRITICAL: Server answer: [21:05:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:06:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:13] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:13] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:06:13] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:13] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:06:13] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:06:14] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:06:14] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:06:14] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:06:14] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:15] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:06:15] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:32] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:06:32] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:33] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:33] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:33] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:33] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:34] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:34] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:34] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:35] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:06:42] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:03] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:07:12] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:07:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:12] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:07:13] ethernet 0/1/7 [damiana bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:07:13] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:22] ethernet 0/1/20 [yucca] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:07:22] ethernet 0/1/11 [thyme] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:07:22] ethernet 0/1/8 [turnera bge1] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:07:22] / on thyme is UNKNOWN: NRPE: Unable to read output [21:07:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:32] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:07:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:07:41] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:07:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:07:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:56] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [21:07:56] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:07:56] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:08:04] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:08:04] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:08:04] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:11:33] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:32] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10858.000000 [21:24:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:24:32] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:24:54] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:25:21] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:25:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:27:22] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:27:55] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:28:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:28:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:29:55] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:30:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:34:55] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:36:03] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:36:32] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [21:39:03] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:40:54] Load avg. on yarrow is WARNING: WARNING - load average: 22.11, 16.44, 8.45 [21:42:22] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:43:13] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [21:44:22] Load avg. on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:44:22] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:13] Load avg. on yarrow is WARNING: WARNING - load average: 26.78, 20.64, 12.01 [21:50:39] nacht ts# [21:50:41] nacht ts [21:54:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:55:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1362979.000000 [21:55:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:58:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 873145.000000 [22:00:54] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 259136 MB (4% inode=62%): [22:02:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2096821.000000 [22:03:32] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:04:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1270307.000000 [22:04:55] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [22:04:55] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:04:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [22:04:55] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:04:55] aliasd on nightshade is CRITICAL: Connection refused [22:04:55] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [22:04:56] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:05:13] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 367146.000000 [22:05:13] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [22:05:13] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:13] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:22] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:05:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48749 MB (7% inode=99%): [22:05:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 597345.000000 [22:05:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:05:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:05:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:55] SSH on mayapple is CRITICAL: Server answer: [22:05:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:06:02] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [22:06:02] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:06:02] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:06:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:16] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:16] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:06:16] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:16] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:06:16] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:06:16] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:16] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:06:16] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:06:16] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:06:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:06:33] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:33] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:33] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:33] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:34] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:34] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:35] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:35] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:35] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:06:42] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:03] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:07:12] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:07:13] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:13] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:13] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:07:13] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:07:13] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:07:14] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:07:14] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:15] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:16] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:07:17] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:32] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:07:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:32] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:07:41] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:07:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:07:54] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:54] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [22:07:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:04] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:04] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:08:04] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:12:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:16:32] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14458.000000 [22:24:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:24:32] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:24:55] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:28:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:29:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:35:03] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:35:33] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [22:54:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:55:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1360971.000000 [22:55:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:58:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 876744.000000 [23:00:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 258831 MB (4% inode=62%): [23:02:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 2099624.000000 [23:04:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1270299.000000 [23:04:55] APT on z-dat-s2-b is WARNING: APT WARNING: 33 packages available for upgrade (0 critical updates). [23:04:55] APT on z-dat-s1-b is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:04:55] APT on yucca is WARNING: APT WARNING: 39 packages available for upgrade (0 critical updates). [23:04:55] APT on nightshade is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:04:55] aliasd on nightshade is CRITICAL: Connection refused [23:04:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [23:04:56] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:05:12] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [23:05:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 366755.000000 [23:05:12] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:13] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:22] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:32] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 48428 MB (7% inode=99%): [23:05:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:05:41] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 593052.000000 [23:05:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:05:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:05:54] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:54] SSH on mayapple is CRITICAL: Server answer: [23:05:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:06:03] APT on sage is WARNING: APT WARNING: 34 packages available for upgrade (0 critical updates). [23:06:04] ethernet 0/1/19 [yucca.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:06:04] ethernet 0/1/6 [turnera bge0] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:06:04] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:13] ethernet 0/1/21 [csw2-esams:2/0/47] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:06:13] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:13] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:13] ethernet 0/1/9 [sage.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:06:14] ethernet 0/1/15 [far1-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:06:14] ethernet 0/1/12 [fsw1-n1-oe16-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:06:20] ethernet 0/1/24 [daphne.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:06:20] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:20] ethernet 0/1/18 [far2-n1-oe16-a-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:06:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:06:32] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:06:42] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:55] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:03] ethernet 0/1/1 [thyme.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:07:12] ethernet 0/1/5 [ts-array5 controller B] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:07:12] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:13] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:13] ethernet 0/1/22 [asw-oe10-esams:0/1/5 LACP primary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:07:13] ethernet 0/1/10 [cassia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:07:13] ethernet 0/1/16 [far1-n1-oe16-b-esams.mgmt] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:07:13] ethernet 0/1/2 [daphne] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:07:14] ethernet 0/1/4 [ts-array5 controller A] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:07:14] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:15] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:07:32] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:32] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:32] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:33] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:33] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:33] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:33] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:34] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:34] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:07:41] ethernet 0/1/23 [asw-oe10-esams:0/1/6 LACP secondary] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:07:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:03] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:03] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:03] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:32] ethernet 0/1/13 [scs-oe16-esams] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:08:33] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:08:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:08:55] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:08:55] ethernet 0/1/3 [adenia] on asw-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host asw-oe16-esams.mgmt. [23:08:55] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:08:55] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:12:32] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:16:32] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18058.000000 [23:23:42] [[Special:Log/newusers]] create 10 * Sadlkfjasdlkj * (New user account) [23:24:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:24:32] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:24:54] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:24:57] [[Special:Log/newusers]] create 10 * Cornelia512 * (New user account) [23:28:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:29:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:34:53] @replag [23:35:02] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:35:32] APT on yarrow is WARNING: APT WARNING: 67 packages available for upgrade (0 critical updates). [23:40:06] @replag [23:40:12] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [23:41:02] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:41:13] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [23:42:06] @replag [23:42:11] @notify tsbot [23:42:11] I will notify you, when I see tsbot around here [23:54:22] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:55:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1359393.000000 [23:55:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:58:55] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 880343.000000