[00:13:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:27:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 714131.000000 [00:39:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:40:25] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105238.000000 [00:43:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:44:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:44:46] / on thyme is UNKNOWN: NRPE: Unable to read output [00:45:06] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:45:26] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [00:46:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:46:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:46:26] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:16] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:17] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:17] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:17] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:17] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:26] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:27] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:36] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:37] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:37] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:37] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:37] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:37] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:38] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:38] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:39] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:39] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:40] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:47:40] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:48:36] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:36] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:36] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:48:46] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:49:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:49:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:26] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1158595.000000 [00:49:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:50:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 541858.000000 [00:50:55] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:05] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27549 MB (4% inode=99%): [00:52:19] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:53:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 844720.000000 [00:55:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:56:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:56:46] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:57:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:57:16] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:57:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:57:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:57:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1426589.000000 [00:57:56] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324945 MB (6% inode=61%): [00:58:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:01:04] K. Peachey * Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [01:13:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:27:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713467.000000 [01:39:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:40:25] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108838.000000 [01:43:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:44:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:44:46] / on thyme is UNKNOWN: NRPE: Unable to read output [01:45:06] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:45:26] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [01:46:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:46:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:46:26] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:47:16] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:16] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:16] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:16] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:26] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:26] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:35] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:36] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:36] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:36] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:37] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:37] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:37] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:37] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:47:38] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:48:36] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:36] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:36] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:36] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:36] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:36] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:36] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:48:46] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:49:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:49:16] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:26] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1154451.000000 [01:49:35] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:50:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 542315.000000 [01:50:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:06] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27484 MB (4% inode=99%): [01:52:16] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:53:25] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 843626.000000 [01:55:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:55:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:56:46] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:57:07] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:57:15] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:57:26] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:57:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:57:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1429969.000000 [01:58:06] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324800 MB (6% inode=61%): [01:58:06] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:13:26] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:27:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 712961.000000 [02:39:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:40:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112437.000000 [02:43:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:44:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:44:46] / on thyme is UNKNOWN: NRPE: Unable to read output [02:45:06] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:45:26] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [02:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:46:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:46:26] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:47:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:16] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:16] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:17] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:17] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:26] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:26] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:36] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:36] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:36] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:36] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:36] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:36] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:36] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:46] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:46] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:47:46] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:48:36] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:36] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:36] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:36] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:37] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:45] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:46] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:48:46] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:49:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:49:16] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:26] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1152856.000000 [02:49:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:50:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 542985.000000 [02:50:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:51:06] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27811 MB (4% inode=99%): [02:52:15] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:53:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 841532.000000 [02:55:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:55:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:56:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:56:46] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:57:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:57:16] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:57:25] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:35] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1433416.000000 [02:58:06] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324332 MB (6% inode=61%): [02:58:06] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:13:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:27:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713039.000000 [03:30:16] Free Memory on damiana is CRITICAL: CRITICAL - 3.0% (253244 kB) free! [03:39:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:40:26] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116038.000000 [03:43:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:44:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:44:46] / on thyme is UNKNOWN: NRPE: Unable to read output [03:45:06] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:45:26] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [03:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:46:06] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:46:16] Free Memory on damiana is WARNING: WARNING - 6.0% (504072 kB) free! [03:46:26] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:16] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:16] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:17] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:17] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:26] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:26] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:36] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:36] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:36] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:36] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:36] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:46] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:46] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:46] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:47:46] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:48:35] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:36] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:36] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:36] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:45] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:46] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:46] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:46] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:48:47] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:49:16] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:26] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1151863.000000 [03:49:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:50:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 543456.000000 [03:50:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:51:06] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27356 MB (4% inode=99%): [03:51:16] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (344644 kB) free! [03:52:16] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:53:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 841298.000000 [03:55:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:55:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:56:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:56:46] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:57:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:57:16] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:57:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:27] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:57:36] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1436859.000000 [03:58:06] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324186 MB (6% inode=61%): [03:58:06] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:13:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:27:27] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 713517.000000 [04:39:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:40:25] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119639.000000 [04:43:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:44:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:44:46] / on thyme is UNKNOWN: NRPE: Unable to read output [04:45:06] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:45:25] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [04:46:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:46:07] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:46:26] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:47:17] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:17] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:17] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:17] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:17] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:26] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:26] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:36] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:36] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:46] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:46] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:46] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:46] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:46] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:46] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:47:47] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:48:35] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:36] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:36] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:46] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:46] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:47] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:47] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:47] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:48:47] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:49:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:49:16] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:26] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1150439.000000 [04:49:36] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:50:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 544037.000000 [04:50:56] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:06] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27303 MB (4% inode=99%): [04:51:26] Free Memory on damiana is CRITICAL: CRITICAL - 1.5% (127252 kB) free! [04:52:16] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:53:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 840174.000000 [04:55:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:55:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:56:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:56:46] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:57:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:57:16] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:57:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:57:27] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:57:35] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1440201.000000 [04:58:06] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324025 MB (6% inode=61%): [04:58:06] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:10:57] FC 0/19 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:10:59] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:10:59] FC 0/16 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:11:00] FC 0/22 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:11:00] FC 0/18 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:11:00] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:13:04] FC 0/21 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:13:37] FC 0/9 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:13:37] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:13:37] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:13:38] FC 0/20 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:13:38] eth0 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:13:38] FC 0/13 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:13:38] FC 0/14 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:13:39] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:18:51] I'm having trouble connecting to the db (specificly enwikinews) on the toolserver... [05:57:09] /mnt user-store on rosemary is CRITICAL: (Service Check Timed Out) [05:59:29] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27279 MB (4% inode=99%): [05:59:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:59:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:59:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:59:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:59:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:59:38] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:59:38] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:59:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:01:47] Free Memory on damiana is OK: OK - 83.7% (7011460 kB) free. [06:03:39] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3148.000000 [06:04:38] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:04:48] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [06:04:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:05:07] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1915.000000 [06:05:08] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3232.000000 [06:05:28] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1839 [06:06:08] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1708.000000 [06:06:28] MySQL slave on rosemary is OK: Uptime: 120837 Threads: 19 Questions: 92925890 Slow queries: 11635 Opens: 8528 Flush tables: 1 Open tables: 2931 Queries per second avg: 769.18 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1649 [06:07:18] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [06:11:38] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3628.000000 [06:12:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3652.000000 [06:58:41] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:58:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:58:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:58:41] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 550533.000000 [06:58:41] / on thyme is UNKNOWN: NRPE: Unable to read output [06:58:41] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 323786 MB (6% inode=61%): [06:58:41] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:42] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [06:58:43] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:58:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:58:49] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [06:58:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:49] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:49] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:49] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:50] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:58:50] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:59:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:08] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:59:08] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:59:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127967.000000 [06:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 720599.000000 [06:59:18] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:59:18] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1156429.000000 [06:59:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:59:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 27010 MB (4% inode=99%): [06:59:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:59:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:59:28] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:59:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:59:40] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:59:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:59:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:59:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:59:48] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:59:58] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 846073.000000 [07:04:40] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [07:04:48] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [07:04:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:07:58] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [07:11:40] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7228.000000 [07:12:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7252.000000 [07:58:41] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:58:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:58:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:58:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 554132.000000 [07:58:42] / on thyme is UNKNOWN: NRPE: Unable to read output [07:58:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 323481 MB (6% inode=61%): [07:58:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:58:42] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:42] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:43] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:43] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:58:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:58:50] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [07:58:50] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:50] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:58:50] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:50] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:51] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:51] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:51] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:58:52] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:59:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:59:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:59:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:59:09] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 131567.000000 [07:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 724199.000000 [07:59:19] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:59:19] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1160029.000000 [07:59:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:59:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26825 MB (4% inode=99%): [07:59:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:59:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:59:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:59:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:59:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:59:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:59:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 849733.000000 [08:00:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:00:41] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:04:40] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [08:04:49] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [08:04:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:40] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [08:11:40] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10828.000000 [08:12:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10852.000000 [08:58:41] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:58:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:58:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:58:41] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 557733.000000 [08:58:41] / on thyme is UNKNOWN: NRPE: Unable to read output [08:58:42] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:42] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 322695 MB (6% inode=61%): [08:58:42] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:58:59] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:59] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:58:59] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:59:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:09] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 135168.000000 [08:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 727799.000000 [08:59:18] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:59:19] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1163629.000000 [08:59:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:59:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26605 MB (4% inode=99%): [08:59:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:59:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:59:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:59:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:59:40] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:40] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:59:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:59:49] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [08:59:49] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:59:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:50] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:51] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:51] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:52] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:52] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:53] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:59:54] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 853333.000000 [09:00:41] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:01:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:04:40] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [09:04:49] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [09:04:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:19] wikidata replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [09:11:40] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14428.000000 [09:12:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14452.000000 [09:57:58] aliasd on yarrow is CRITICAL: Connection refused [09:58:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:59:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:59:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 730456.000000 [09:59:18] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:59:28] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1166371.000000 [09:59:29] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138779.000000 [09:59:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:59:29] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26339 MB (4% inode=99%): [09:59:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:59:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:59:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:59:30] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:59:41] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:59:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:59:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:59:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:59:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:59:42] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 560346.000000 [09:59:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:59:43] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 322320 MB (6% inode=61%): [09:59:43] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:59:44] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:59:44] / on thyme is UNKNOWN: NRPE: Unable to read output [09:59:45] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:59:45] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [09:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:59:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:59:49] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 38811 [09:59:49] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:59:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:08] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:00:08] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:00:09] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 855898.000000 [10:00:41] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:00:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:00:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:00:49] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:00:49] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:00:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:00:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:00:50] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:01:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:04:18] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 4.327 second response time [10:04:40] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16579 [10:04:53] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10335 [10:04:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:05:08] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.004 second response time [10:09:17] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16833.000000 [10:11:49] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15970.000000 [10:12:08] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4569.000000 [10:15:08] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3365.000000 [10:19:08] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1783.000000 [10:20:19] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 9.606 second response time [10:44:19] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 104926 MB (10% inode=99%): [10:57:58] aliasd on yarrow is CRITICAL: Connection refused [10:59:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:59:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 730791.000000 [10:59:18] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:59:28] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1165346.000000 [10:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:59:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:59:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:59:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:59:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:59:41] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 560185.000000 [10:59:41] / on thyme is UNKNOWN: NRPE: Unable to read output [10:59:42] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:59:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321945 MB (6% inode=61%): [10:59:43] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:59:43] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:59:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [10:59:44] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:59:48] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:59:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:59:49] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25997 [10:59:50] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:59:50] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:00:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:00:09] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 858379.000000 [11:00:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 26087 MB (4% inode=99%): [11:00:28] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 142441.000000 [11:00:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:00:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:00:28] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:00:41] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:00:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:00:41] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:00:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:00:49] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:00:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:00:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:00:49] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:00:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:00:50] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:00:51] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:00:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1461476.000000 [11:01:40] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:04:40] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18025 [11:04:48] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10266 [11:04:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:18] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18147.000000 [11:11:48] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11524.000000 [11:32:44] @replag [11:32:45] Pyfisch: s1-rr-a: 4h 14m 5s [-]; s1-rr-a-wd: 1w 1d 7h 51m 21s [-]; s1-user-c: 1d 16h 6m 26s [-]; s1-user-wd: 1w 6d 11h 36m 50s [-]; s2-rr: 2h 44m 3s [-]; s2-user: 2h 44m 3s [-]; s2-user-c: error; s2-user-wd: 1w 2d 22h 54m 3s [-] [11:32:46] Pyfisch: s3-user: 17s [-]; s3-user-wd: 1w 1d 22h 42m 42s [-]; s4-user-wd: 2w 2d 22h 25m 48s [-]; s5-rr-a-wd: 6d 11h 36m 19s [-]; s5-user: 4h 42m 50s [-]; s5-user-c: 2h 18m 58s [-]; s6-user-wd: 1w 2d 16h 30m 9s [-]; s7-user-wd: 1w 1d 10h 59m 18s [-] [11:34:40] are there plans to reduce the replication lag, or are you waiting for labs having db access? [11:35:08] editcounters are nearly unusable with lags higher than one week [11:45:05] Dr. Trigon * [Toolserver-l] Cronjobs [11:45:05] Dr. Trigon * [Toolserver-l] Doxygen and coverage [11:47:49] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3415.000000 [11:51:49] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1691.000000 [11:57:58] aliasd on yarrow is CRITICAL: Connection refused [11:59:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:59:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731022.000000 [11:59:18] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:59:28] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1163795.000000 [11:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:59:50] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:59:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:59:50] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4196 [11:59:50] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:50] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:59:50] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:59:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:59:50] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:59:51] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:59:51] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:59:52] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:59:58] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:59:59] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:59:59] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:19] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 861454.000000 [12:00:28] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 146040.000000 [12:00:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 25865 MB (4% inode=99%): [12:00:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:00:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:00:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:00:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:00:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:00:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:00:41] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 560356.000000 [12:00:41] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:41] / on thyme is UNKNOWN: NRPE: Unable to read output [12:00:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:41] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321472 MB (6% inode=61%): [12:00:42] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:42] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [12:00:43] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:00:49] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:49] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:49] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:00:49] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:58] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:58] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:58] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:58] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:58] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:00:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1464667.000000 [12:01:08] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:01:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:41] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:01:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:01:49] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3451 [12:02:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:04:40] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12288 [12:04:49] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7277 [12:04:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:18] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11709.000000 [12:10:49] MySQL slave on z-dat-s1-b is OK: Uptime: 9262 Threads: 11 Questions: 4633935 Slow queries: 112 Opens: 218 Flush tables: 1 Open tables: 181 Queries per second avg: 500.316 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1657 [12:24:41] / on wolfsbane is WARNING: DISK WARNING - free space: / 6211 MB (20% inode=93%): [12:24:49] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6211 MB (20% inode=93%): [12:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:34:48] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3321 [12:37:49] MySQL slave on z-dat-s2-b is OK: Uptime: 10924 Threads: 13 Questions: 11743868 Slow queries: 256 Opens: 95463 Flush tables: 1 Open tables: 256 Queries per second avg: 1075.51 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1767 [12:47:29] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [12:50:08] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1322.000000 [12:57:59] aliasd on yarrow is CRITICAL: Connection refused [12:59:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731387.000000 [12:59:19] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:59:41] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:59:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:59:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1165002.000000 [12:59:49] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:59:49] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:59:50] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:59:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:59:50] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:59:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:59:59] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:59:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:59:59] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:59:59] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 864418.000000 [13:00:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 25559 MB (4% inode=99%): [13:00:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:00:28] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:00:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:00:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:00:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 560717.000000 [13:00:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:41] / on thyme is UNKNOWN: NRPE: Unable to read output [13:00:41] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321150 MB (6% inode=61%): [13:00:42] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:43] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:00:44] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:00:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 149662.000000 [13:00:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:50] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:50] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:50] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:00:50] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:59] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:59] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:59] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:00:59] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:01:08] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:01:09] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:01:28] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:01:41] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:01:41] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:01:49] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:01:58] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:01:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1468110.000000 [13:02:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:04:28] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2170 [13:04:40] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13531 [13:04:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:04:58] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2193.000000 [13:09:18] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13617.000000 [13:23:28] MySQL slave on rosemary is OK: Uptime: 147057 Threads: 9 Questions: 106266764 Slow queries: 18074 Opens: 9952 Flush tables: 1 Open tables: 3078 Queries per second avg: 722.622 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1708 [13:23:58] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1639.000000 [13:24:49] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5470 MB (18% inode=93%): [13:25:40] / on wolfsbane is WARNING: DISK WARNING - free space: / 5456 MB (18% inode=93%): [13:29:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:55:41] / on wolfsbane is OK: DISK OK - free space: / 13612 MB (45% inode=93%): [13:55:49] /tmp on wolfsbane is OK: DISK OK - free space: / 13611 MB (45% inode=93%): [13:56:58] aliasd on yarrow is OK: TCP OK - 0.008 second response time on port 984 [500 Not found.] [13:59:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:59:19] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731773.000000 [13:59:19] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:59:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1164203.000000 [13:59:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:59:49] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:49] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:59:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:58] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:59:58] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:59:58] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:59:58] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 867300.000000 [14:00:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 25285 MB (4% inode=99%): [14:00:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:00:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:00:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:00:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:00:41] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 560270.000000 [14:00:41] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:41] / on thyme is UNKNOWN: NRPE: Unable to read output [14:00:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320796 MB (6% inode=61%): [14:00:42] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:43] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:00:44] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:00:48] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 153263.000000 [14:00:49] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:59] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:59] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:59] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:00:59] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:59] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:00:59] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:01:00] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:01:00] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:01:01] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:01:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:01:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:01:09] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:28] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:01:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:01:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:01:40] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:01:49] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:01:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1471499.000000 [14:02:29] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:02:59] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:58] aliasd on yarrow is CRITICAL: Connection refused [14:04:41] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7721 [14:04:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:19] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7425.000000 [14:26:49] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3541 [14:27:19] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3376.000000 [14:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:34:49] MySQL slave on z-dat-s5-b is OK: Uptime: 17874 Threads: 5 Questions: 91482039 Slow queries: 222 Opens: 1165 Flush tables: 1 Open tables: 255 Queries per second avg: 5118.162 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1734 [14:35:18] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1697.000000 [14:59:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731921.000000 [14:59:18] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:59:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:59:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1160830.000000 [14:59:58] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:58] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:59:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:59:59] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:59:59] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:59:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:59:59] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 870067.000000 [15:00:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 25044 MB (4% inode=99%): [15:00:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:00:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 156865.000000 [15:00:59] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:00:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:00:59] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:59] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:59] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:59] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:00:59] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:00] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:00] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:01] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:01] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:02] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:09] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:01:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:01:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:01:40] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:01:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:01:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:01:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 559457.000000 [15:01:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:01:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:01:41] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:42] / on thyme is UNKNOWN: NRPE: Unable to read output [15:01:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320399 MB (6% inode=61%): [15:01:43] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:43] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:44] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:01:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:01:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1474868.000000 [15:01:58] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:02:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:02:59] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:58] aliasd on yarrow is CRITICAL: Connection refused [15:04:59] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:59:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731791.000000 [15:59:18] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:59:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:59:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:59:50] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1159643.000000 [15:59:58] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:58] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:59] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:59:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:59:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:59:59] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:59:59] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:19] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 872973.000000 [16:00:29] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 24722 MB (4% inode=99%): [16:00:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:00:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 160465.000000 [16:00:59] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:00:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:00:59] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:59] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:59] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:59] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:00:59] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:01:00] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:01:00] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:01:01] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:01:01] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:01:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:01:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:01:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:01:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:01:40] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:01:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:01:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:01:41] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:01:41] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 558372.000000 [16:01:41] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:01:41] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:01:42] / on thyme is UNKNOWN: NRPE: Unable to read output [16:01:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 319983 MB (6% inode=61%): [16:01:43] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:01:43] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:01:44] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:01:44] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:01:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1478272.000000 [16:02:40] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:02:58] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:02:58] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:58] aliasd on yarrow is CRITICAL: Connection refused [16:04:58] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:57:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319533 MB (5% inode=61%): [16:59:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731555.000000 [16:59:18] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:59:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:59:48] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:59:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:59:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1157609.000000 [17:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 875881.000000 [17:00:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 24524 MB (4% inode=99%): [17:00:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:00:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 164065.000000 [17:00:59] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:00:59] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:59] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:00:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:00] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:01:00] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:01:00] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:01:01] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:01:01] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:01:02] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:01:02] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:01:03] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:01:03] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:01:04] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:01:28] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:01:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:01:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:01:40] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:01:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:01:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:01:41] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 557335.000000 [17:01:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:01:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1481699.000000 [17:02:08] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:02:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:02:09] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:02:39] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:02:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:02:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:02:40] / on thyme is UNKNOWN: NRPE: Unable to read output [17:02:40] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:02:40] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:02:41] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:02:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [17:02:58] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:02:58] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:05:01] aliasd on yarrow is CRITICAL: Connection refused [17:05:01] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:36:18] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.608 second response time [17:37:09] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.997 second response time [17:38:19] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.211 second response time [17:57:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 319157 MB (5% inode=61%): [17:59:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731279.000000 [17:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:59:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:59:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1157566.000000 [18:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 878784.000000 [18:00:18] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:00:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 24174 MB (3% inode=99%): [18:00:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:00:50] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 167667.000000 [18:01:08] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:10] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:10] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:10] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:10] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:11] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:11] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:01:12] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:01:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:01:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:01:40] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:01:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:01:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:01:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556118.000000 [18:01:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:01:59] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:01:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1485106.000000 [18:01:59] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:59] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:00] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:00] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:01] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:09] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:09] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:02:39] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:02:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:02:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:40] / on thyme is UNKNOWN: NRPE: Unable to read output [18:02:40] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:41] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:41] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:03:09] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:03:58] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:59] aliasd on yarrow is CRITICAL: Connection refused [18:04:59] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:56:58] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2029.000000 [18:57:28] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2035 [18:57:39] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318681 MB (5% inode=61%): [18:59:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 731539.000000 [18:59:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:59:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:59:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1158761.000000 [19:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 881731.000000 [19:00:19] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:00:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 23908 MB (3% inode=99%): [19:00:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:00:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 171267.000000 [19:01:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:01:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:01:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:01:40] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:01:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:01:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:01:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555434.000000 [19:01:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:01:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1488584.000000 [19:01:59] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:01:59] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:59] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:01:59] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:09] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:09] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:09] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:10] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:11] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:12] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:12] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:13] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:13] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:14] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:39] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:02:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:02:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:40] / on thyme is UNKNOWN: NRPE: Unable to read output [19:02:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:41] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:41] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:03:09] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:03:58] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:59] aliasd on yarrow is CRITICAL: Connection refused [19:05:00] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:49] / on wolfsbane is WARNING: DISK WARNING - free space: / 6118 MB (20% inode=93%): [19:05:50] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6118 MB (20% inode=93%): [19:09:58] aliasd on yarrow is OK: TCP OK - 0.008 second response time on port 984 [500 Not found.] [19:13:49] / on wolfsbane is OK: DISK OK - free space: / 13470 MB (44% inode=93%): [19:13:49] /tmp on wolfsbane is OK: DISK OK - free space: / 13470 MB (44% inode=93%): [19:21:28] MySQL slave on rosemary is OK: Uptime: 168537 Threads: 12 Questions: 125218949 Slow queries: 23329 Opens: 10729 Flush tables: 1 Open tables: 3087 Queries per second avg: 742.976 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1780 [19:21:58] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1795.000000 [19:28:28] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1840 [19:28:59] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1824.000000 [19:29:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:30:58] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1780.000000 [19:31:28] MySQL slave on rosemary is OK: Uptime: 169138 Threads: 12 Questions: 125712634 Slow queries: 23457 Opens: 10745 Flush tables: 1 Open tables: 3086 Queries per second avg: 743.254 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1737 [19:39:04] Platonides * Re: [Toolserver-l] Doxygen and coverage [19:57:39] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318328 MB (5% inode=61%): [19:59:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 732675.000000 [19:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:59:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:59:50] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1158260.000000 [20:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 884722.000000 [20:00:19] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:00:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 23462 MB (3% inode=99%): [20:00:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:00:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 174867.000000 [20:01:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:01:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:01:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:01:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:01:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:01:40] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:01:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555159.000000 [20:01:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:01:41] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:01:59] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1491940.000000 [20:01:59] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:02:09] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:09] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:09] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:10] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:11] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:12] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:02:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:41] / on thyme is UNKNOWN: NRPE: Unable to read output [20:02:41] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:02:41] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:02:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:50] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:09] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:09] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:09] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:09] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:09] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:10] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:11] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:12] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:12] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:05:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:43:04] Maarten Dammers * Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [20:57:40] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 317997 MB (5% inode=61%): [20:59:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:59:18] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 733611.000000 [20:59:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:59:49] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:59:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:59:50] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1157954.000000 [21:00:18] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 887771.000000 [21:00:28] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 23250 MB (3% inode=99%): [21:00:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:00:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 178467.000000 [21:01:18] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:01:40] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:01:40] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:01:40] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555225.000000 [21:01:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:01:40] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:02:09] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:10] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:10] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:10] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:10] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:10] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:11] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:11] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:12] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:29] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:02:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:29] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:02:29] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:02:40] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:02:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:40] / on thyme is UNKNOWN: NRPE: Unable to read output [21:02:40] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:02:40] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:02:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:49] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1495427.000000 [21:02:58] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:04:09] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:09] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:10] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:10] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:10] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:11] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:12] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:04:12] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:04:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:09] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:38:59] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2222.000000 [21:48:18] Free Memory on turnera is WARNING: WARNING - 5.7% (478560 kB) free! [21:52:04] Marc A. Pelletier * Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [21:57:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 317700 MB (5% inode=61%): [21:59:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:59:19] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 734412.000000 [21:59:19] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (398336 kB) free! [21:59:50] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:59:50] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:59:50] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1157307.000000 [22:01:50] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:01:50] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 555952.000000 [22:01:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:01:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:01:59] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3602.000000 [22:01:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:02:14] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:14] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:14] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:14] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:02:14] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:02:15] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:02:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:02:28] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:02:44] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:48] / on thyme is UNKNOWN: NRPE: Unable to read output [22:02:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:02:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:02:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:02:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:02:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1498846.000000 [22:02:58] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:02:58] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:02:58] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:02:59] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:03:08] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:03:08] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:03:28] Free Memory on turnera is OK: OK - 87.8% (7360376 kB) free. [22:05:07] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:08] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:09] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:58] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3593.000000 [22:06:08] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:57] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3548 [22:26:58] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1637.000000 [22:26:58] MySQL slave on rosemary is OK: Uptime: 179664 Threads: 6 Questions: 133248591 Slow queries: 25838 Opens: 11560 Flush tables: 1 Open tables: 3089 Queries per second avg: 741.654 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1633 [22:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:36:18] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2170.000000 [22:36:58] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2213.000000 [22:44:17] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 105123 MB (10% inode=99%): [22:58:49] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 317348 MB (5% inode=61%): [22:59:48] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 738007.000000 [23:00:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:00:28] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:00:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1160856.000000 [23:00:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:01:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:01:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 894540.000000 [23:01:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:01:49] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 559413.000000 [23:01:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:01:49] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 185728.000000 [23:01:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 22628 MB (3% inode=99%): [23:01:58] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:02:07] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:08] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:08] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:09] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:09] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:11] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:11] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:12] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:02:28] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:02:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:02:48] / on thyme is UNKNOWN: NRPE: Unable to read output [23:02:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:02:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:02:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:02:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:02:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:02:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:02:58] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1727.000000 [23:02:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1502033.000000 [23:03:08] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:03:08] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:03:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:03:08] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:03:08] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:03:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:09] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:03:11] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:03:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:03:12] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:03:37] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:08] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:05:08] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:05:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:05:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:17] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1430.000000 [23:06:08] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:04] DaB. * Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [23:23:04] Marc A. Pelletier * Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [23:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:58:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 316000 MB (5% inode=61%): [23:59:48] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 738126.000000