[00:26:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:26:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:26:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:26:05] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 278711 MB (5% inode=62%): [00:26:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:26:06] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:26:15] APT on sage is CRITICAL: APT CRITICAL: 7 packages available for upgrade (4 critical updates). [00:26:15] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:15] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:25] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:25] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:26] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:26:26] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 872697.000000 [00:26:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:26] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:26:27] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:26:27] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:28] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:34] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:26:35] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:35] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:36] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:44] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 609078.000000 [00:26:45] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 136693.000000 [00:26:46] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 633986.000000 [00:26:46] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:46] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:46] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:46] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:26:46] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 215568.000000 [00:26:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67514 MB (11% inode=99%): [00:26:56] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:26:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:26:56] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:56] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:26:56] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:26:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:27:05] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [00:27:05] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:05] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:06] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:06] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:06] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:15] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:27:35] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:35] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:45] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1018308.000000 [00:27:45] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:27:55] / on thyme is UNKNOWN: NRPE: Unable to read output [00:28:05] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:28:05] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:28:05] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:28:05] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [00:32:15] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:32:24] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:32:35] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:35] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39864.000000 [00:32:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39863.000000 [00:32:35] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 487964.000000 [00:56:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:26:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:26:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:26:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:26:05] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 278609 MB (5% inode=62%): [01:26:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:26:05] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:26:15] APT on sage is CRITICAL: APT CRITICAL: 7 packages available for upgrade (4 critical updates). [01:26:15] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:15] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:15] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:25] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:25] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:26] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:26:26] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:26:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 876297.000000 [01:26:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:26:26] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:26:27] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:26:27] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:28] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:35] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:35] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:36] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:26:44] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 140293.000000 [01:26:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 612678.000000 [01:26:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 637585.000000 [01:26:45] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:26:45] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:46] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:46] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:26:46] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 219167.000000 [01:26:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67345 MB (11% inode=99%): [01:26:55] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:26:56] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:26:56] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:56] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:26:56] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:26:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:27:04] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [01:27:05] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:05] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:05] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:06] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:06] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:15] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:27:35] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:35] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:45] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1021907.000000 [01:27:45] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:27:55] / on thyme is UNKNOWN: NRPE: Unable to read output [01:28:05] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:28:05] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:28:05] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:28:05] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [01:32:14] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:32:24] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:32:35] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:35] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43464.000000 [01:32:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43463.000000 [01:32:35] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 491564.000000 [01:53:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66999 MB (10% inode=99%): [01:56:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:26:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:26:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:26:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:26:05] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 278485 MB (5% inode=62%): [02:26:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:26:06] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:26:15] APT on sage is CRITICAL: APT CRITICAL: 7 packages available for upgrade (4 critical updates). [02:26:15] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:15] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:25] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:26] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:26:26] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 879898.000000 [02:26:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:26] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:26:26] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:26:27] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:27] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:35] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:35] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:35] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:26:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616278.000000 [02:26:45] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 143893.000000 [02:26:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 641185.000000 [02:26:46] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:46] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:46] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:46] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:26:46] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 222768.000000 [02:26:55] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:26:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:26:55] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:55] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:26:55] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:26:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:27:05] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [02:27:05] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:05] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:05] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:05] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:05] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:15] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:35] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:35] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:27:45] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1025508.000000 [02:27:45] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:27:55] / on thyme is UNKNOWN: NRPE: Unable to read output [02:28:05] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:28:06] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:28:06] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:28:06] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [02:32:15] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:32:25] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:32:35] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:35] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47064.000000 [02:32:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47063.000000 [02:32:35] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:34:15] Free Memory on damiana is WARNING: WARNING - 6.9% (575404 kB) free! [02:34:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 495165.000000 [02:42:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67815 MB (11% inode=99%): [02:51:14] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (344368 kB) free! [02:56:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:17:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67040 MB (10% inode=99%): [03:26:06] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:26:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:26:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:26:08] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 278400 MB (5% inode=62%): [03:26:08] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:26:08] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:26:16] APT on sage is CRITICAL: APT CRITICAL: 7 packages available for upgrade (4 critical updates). [03:26:16] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:16] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:16] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:24] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:26:25] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 883498.000000 [03:26:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:26] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:26] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:26] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:26:26] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:26:27] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:27] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:35] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:35] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:35] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:36] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:26:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 619878.000000 [03:26:45] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 147493.000000 [03:26:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 644785.000000 [03:26:45] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:45] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:46] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:46] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:26:46] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 226368.000000 [03:26:55] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:26:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:26:55] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:56] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:26:56] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:26:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:27:12] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [03:27:26] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:29:30] MySQL slave on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [03:29:31] Environment IPMI on thyme is CRITICAL: (Service Check Timed Out) [03:29:32] Load avg. on thyme is CRITICAL: (Service Check Timed Out) [03:29:33] SMTP on yucca is CRITICAL: (Service Check Timed Out) [03:33:01] / on thyme is CRITICAL: (Service Check Timed Out) [03:37:47] /mnt on thyme is CRITICAL: (Service Check Timed Out) [03:38:02] wikidata replag on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [03:38:15] /tmp on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:40:52] qstat prints error: error: getpwuid(8396) failed: Error 0 [03:40:54] thoughts? [03:41:11] addshore, Krinkle|detached? [03:41:22] Merlissimo: ^ [03:43:23] Hmm, i think maybe toolsever ssh is just down. [03:43:34] ^^ points up: " /mnt on thyme is CRITICAL: (Service Check Timed Out)" [03:43:39] that would make sense [03:44:18] been logging weird stuff in this channel since ~21.45 when the bot began doing the same thing over...and over...etc [03:47:38] @replag [03:48:01] And "Known issues: SSH, CRON" [03:52:58] oh yeah [06:43:04] Theopolisme * [Toolserver-l] Qcronsub and qstat trickery [07:09:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 239729.000000 [07:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65858 MB (10% inode=99%): [07:09:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:09:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:38] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:09:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:09:39] / on thyme is UNKNOWN: NRPE: Unable to read output [07:09:47] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:48] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:48] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:48] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [07:09:48] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:48] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:48] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:09:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:14:37] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:14:56] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [07:14:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [07:15:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:15:48] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:15:48] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:08:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:08:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:08:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:08:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 277442 MB (5% inode=62%): [08:08:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:08:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:08:48] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:08:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:08:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 156349 [08:08:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:08:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:08:57] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [08:08:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:08:57] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:08:57] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:09:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 900338.000000 [08:09:07] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:07] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:07] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:09:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60580.000000 [08:09:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55463.000000 [08:09:18] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:18] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:19] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 156370.000000 [08:09:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 627965.000000 [08:09:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:09:27] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1039979.000000 [08:09:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 242689.000000 [08:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65555 MB (10% inode=99%): [08:09:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:09:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:38] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:09:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:09:39] / on thyme is UNKNOWN: NRPE: Unable to read output [08:09:47] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:47] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:47] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:47] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [08:09:47] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:48] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:48] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:09:56] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:10:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:10:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:10:07] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:10:07] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:10:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:11:04] Artem Korzhimanov * Re: [Toolserver-l] Qcronsub and qstat trickery [08:14:37] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:14:56] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [08:15:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:15:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:15:48] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:15:57] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [09:08:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:08:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:08:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:08:56] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 148866 [09:08:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:08:57] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [09:08:57] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:08:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:08:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:08:57] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:09:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 903835.000000 [09:09:07] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:07] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:07] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 148726.000000 [09:09:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:09:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 622593.000000 [09:09:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:09:27] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:09:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1039860.000000 [09:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 245608.000000 [09:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65246 MB (10% inode=99%): [09:09:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:09:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:38] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:09:39] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:09:39] / on thyme is UNKNOWN: NRPE: Unable to read output [09:09:47] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:47] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:47] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:09:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 276979 MB (5% inode=62%): [09:09:47] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [09:09:47] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:48] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:09:56] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:10:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:10:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:10:16] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:10:16] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:10:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:10:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52275.000000 [09:10:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45392.000000 [09:10:18] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:18] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:19] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:48] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:10:48] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:10:48] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:11:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:11:06] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:11:07] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:11:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:14:37] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:14:56] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [09:15:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:15:47] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:15:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [09:16:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:38:27] s4 replag on rosemary is CRITICAL: (Service Check Timed Out) [09:39:37] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2209 [09:40:47] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [09:41:06] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2284.000000 [09:41:16] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2387.000000 [10:02:57] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3618.000000 [10:06:56] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [10:08:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:08:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:08:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:09:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 907317.000000 [10:09:07] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:07] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:07] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:09:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 618083.000000 [10:09:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 144163.000000 [10:09:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:09:27] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:09:38] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:38] wikidata replag on rosemary is CRITICAL: (Service Check Timed Out) [10:09:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 248460.000000 [10:09:38] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64987 MB (10% inode=99%): [10:09:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:09:38] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:38] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:39] / on thyme is UNKNOWN: NRPE: Unable to read output [10:09:40] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:09:47] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:47] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:47] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:09:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 276559 MB (5% inode=62%): [10:09:48] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:48] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:48] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [10:09:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:09:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 144194 [10:09:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:09:57] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [10:09:57] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:09:58] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:10:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:10:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:10:37] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3620 [10:10:47] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:10:47] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:10:48] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:11:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:11:07] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:11:07] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:11:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:11:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:11:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:11:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:11:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:11:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48963.000000 [10:11:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34379.000000 [10:11:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:11:18] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:14:36] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:14:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [10:15:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:15:47] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:15:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [10:16:36] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:27:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 18345 MB (1% inode=99%): [11:03:56] s4 replag on rosemary is CRITICAL: (Service Check Timed Out) [11:07:46] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [11:08:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:08:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:09:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 910736.000000 [11:09:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:06] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:06] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:07] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:26] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 618613.000000 [11:09:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 139130.000000 [11:09:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:09:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:09:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 251240.000000 [11:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64557 MB (10% inode=99%): [11:09:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:09:48] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:48] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:48] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:09:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 276162 MB (5% inode=62%): [11:09:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:09:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:49] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [11:09:50] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:09:56] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:09:56] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 139093 [11:09:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:10:06] wikidata replag on rosemary is CRITICAL: (Service Check Timed Out) [11:10:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:10:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:36] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:10:36] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:10:37] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:10:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:10:37] / on thyme is UNKNOWN: NRPE: Unable to read output [11:10:37] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7141 [11:10:47] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:10:56] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [11:10:56] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:10:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:10:57] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:10:57] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:11:05] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:11:06] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:11:07] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:11:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:11:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:11:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:11:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:11:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:11:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43627.000000 [11:11:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22409.000000 [11:11:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:18] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:11:47] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:11:47] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:14:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [11:15:37] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:15:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:15:47] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:16:36] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:16:57] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [11:52:02] hi [11:52:04] what about my request? https://jira.toolserver.org/browse/ACCAPP-632 [11:52:29] it was created month ago [11:52:37] and not fixed yet [12:04:29] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10866.000000 [12:08:37] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [12:08:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:08:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:09:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 914166.000000 [12:09:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:06] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:06] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:09:36] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 254187.000000 [12:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64293 MB (10% inode=99%): [12:09:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:09:47] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:09:47] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:09:47] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:09:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:09:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 275715 MB (5% inode=62%): [12:09:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:09:48] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:09:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:09:49] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [12:09:50] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:09:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:09:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 135624 [12:09:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:10:07] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:10:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616529.000000 [12:10:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 135653.000000 [12:10:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:10:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:10:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:37] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:10:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:10:37] / on thyme is UNKNOWN: NRPE: Unable to read output [12:10:37] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10733 [12:10:47] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:56] wikidata replag on rosemary is CRITICAL: (Service Check Timed Out) [12:10:56] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [12:10:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:57] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:58] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:10:58] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:11:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:11:06] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:11:06] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:11:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:11:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:11:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:11:47] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:11:47] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:12:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:12:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:12:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:12:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36897.000000 [12:12:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14334.000000 [12:12:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:12:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:14:56] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [12:15:37] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:15:48] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:15:48] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:16:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [12:17:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:20:22] Ahonc: it takes time [13:04:47] s4 replag on rosemary is CRITICAL: (Service Check Timed Out) [13:08:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:08:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:09:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 917580.000000 [13:09:07] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:07] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:09:16] s1 replag on rosemary is CRITICAL: (Service Check Timed Out) [13:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 257229.000000 [13:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63980 MB (10% inode=99%): [13:09:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:09:47] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:09:47] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:09:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:09:56] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 132218 [13:09:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:10:06] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:10:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:10:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:10:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:10:37] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:10:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:10:37] / on thyme is UNKNOWN: NRPE: Unable to read output [13:10:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13914 [13:10:47] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:10:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:10:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 275029 MB (5% inode=62%): [13:10:48] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:10:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:10:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:10:48] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:10:48] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [13:10:48] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:07] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:11:07] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:11:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:11:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:26] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:11:26] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:26] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 131887.000000 [13:11:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616665.000000 [13:11:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:11:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:37] wikidata replag on rosemary is CRITICAL: (Service Check Timed Out) [13:11:47] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:47] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:56] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [13:11:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:57] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:11:57] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:12:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:12:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:12:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32183.000000 [13:12:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10016.000000 [13:12:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:12:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:13:16] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:15:36] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:15:48] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:15:48] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:15:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [13:17:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:17:57] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [14:05:27] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9535.000000 [14:08:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:08:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:09:07] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:09:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 921017.000000 [14:09:07] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:07] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:09:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11192.000000 [14:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 260335.000000 [14:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63704 MB (10% inode=99%): [14:09:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:09:47] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:09:47] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:10:06] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:10:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:10:38] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:10:38] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:10:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:10:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:10:38] / on thyme is UNKNOWN: NRPE: Unable to read output [14:10:38] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10992 [14:10:48] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:10:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:10:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 274590 MB (5% inode=62%): [14:10:48] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:10:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:10:48] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:10:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:10:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:10:49] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [14:10:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:10:56] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 122336 [14:10:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:11:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:11:07] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:11:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:11:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:11:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:11:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122274.000000 [14:11:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 616437.000000 [14:11:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:11:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:11:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:11:47] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:11:47] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:11:56] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [14:11:56] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:12:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:12:06] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:12:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:12:27] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1055682.000000 [14:12:56] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:12:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:12:57] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:13:16] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:13:16] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:13:16] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25860.000000 [14:13:16] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5144.000000 [14:13:16] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:38] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:15:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:15:48] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:16:00] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [14:17:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:17:57] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [14:39:17] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3535.000000 [14:46:27] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3577.000000 [15:03:04] Theopolisme * Re: [Toolserver-l] Qcronsub and qstat trickery [15:06:27] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000 [15:07:04] Wolfgang Faust * Re: [Toolserver-l] Qcronsub and qstat trickery [15:09:07] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:09:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 924471.000000 [15:09:17] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5525.000000 [15:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:09:38] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:09:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 263536.000000 [15:09:38] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63181 MB (10% inode=99%): [15:09:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:09:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:09:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:09:48] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:09:49] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:10:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:06] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:10:06] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:10:06] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:10:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:10:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:10:38] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:10:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:10:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:10:38] / on thyme is UNKNOWN: NRPE: Unable to read output [15:10:47] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:10:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:10:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:10:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 110443 [15:10:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:11:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:11:07] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:11:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:11:37] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5438 [15:11:47] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:11:47] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:11:47] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:11:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 274158 MB (5% inode=62%): [15:11:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:11:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:11:48] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:11:48] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:11:49] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [15:11:56] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [15:11:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:12:06] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:12:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:12:16] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:12:26] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1057229.000000 [15:12:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:12:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:12:28] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109850.000000 [15:12:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 617555.000000 [15:12:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:12:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:12:29] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:12:57] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:12:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:13:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:13:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:13:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25058.000000 [15:13:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:56] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:15:37] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:15:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:15:47] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:16:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [15:17:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:17:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [15:22:04] Theopolisme * Re: [Toolserver-l] Qcronsub and qstat trickery [15:36:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3634.000000 [16:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:09:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 928068.000000 [16:09:16] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5697.000000 [16:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:09:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 267126.000000 [16:09:38] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62919 MB (10% inode=99%): [16:09:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:09:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:09:48] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:09:48] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:10:07] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:07] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:10:07] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:10:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:10:37] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:10:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:10:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:10:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:10:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:10:37] / on thyme is UNKNOWN: NRPE: Unable to read output [16:10:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:10:47] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:10:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:10:56] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 93106 [16:11:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:11:06] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:11:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:11:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:11:37] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5841 [16:11:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 273751 MB (5% inode=62%): [16:11:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:11:48] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:11:48] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:11:48] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:11:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:11:57] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [16:11:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:11:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:12:06] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:12:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:12:27] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1059803.000000 [16:12:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:12:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:12:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:12:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 92596.000000 [16:12:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 621150.000000 [16:12:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:12:29] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:12:29] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:12:47] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:12:47] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:12:48] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [16:12:56] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:13:16] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:13:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:13:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:13:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28658.000000 [16:13:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:13:57] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:15:37] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:16:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:16:48] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:16:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [16:17:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:17:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [16:21:27] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2164.000000 [16:27:04] Platonides * Re: [Toolserver-l] Qcronsub and qstat trickery [16:27:27] s4 replag on rosemary is CRITICAL: (Service Check Timed Out) [16:36:16] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7235.000000 [17:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:09:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 931667.000000 [17:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:09:38] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:09:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 270727.000000 [17:09:38] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62658 MB (10% inode=99%): [17:09:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:09:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:09:54] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:09:54] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:10:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8815.000000 [17:10:07] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:07] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:10:07] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:10:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:10:37] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:10:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:10:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:10:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:10:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:10:37] / on thyme is UNKNOWN: NRPE: Unable to read output [17:10:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:10:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:10:56] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 75328 [17:11:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:11:06] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:11:36] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8777 [17:11:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 273312 MB (5% inode=62%): [17:11:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:11:48] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:11:48] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:11:48] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:11:48] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:11:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:11:57] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [17:11:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:11:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:12:07] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:12:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:12:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:12:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:12:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:12:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:12:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74853.000000 [17:12:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:12:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 624750.000000 [17:12:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:12:29] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:12:47] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [17:12:47] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:12:47] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:12:56] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:13:16] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3433.000000 [17:13:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1063272.000000 [17:13:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:13:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:13:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:13:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32258.000000 [17:13:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:18] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:13:57] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:13:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:15:37] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:16:48] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:16:48] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:16:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [17:17:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:17:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [17:23:04] Theopolisme * Re: [Toolserver-l] Qcronsub and qstat trickery [17:28:17] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1757.000000 [17:37:16] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10894.000000 [18:09:07] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 274326.000000 [18:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62401 MB (10% inode=99%): [18:09:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:09:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:10:06] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6427.000000 [18:10:07] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 935327.000000 [18:10:07] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:07] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:10:38] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:10:38] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:10:38] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:10:38] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:10:38] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:10:38] / on thyme is UNKNOWN: NRPE: Unable to read output [18:10:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:10:48] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:10:48] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:10:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:10:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 56285 [18:11:06] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:11:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:11:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:11:06] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:11:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6363 [18:11:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 272672 MB (5% inode=62%): [18:11:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:11:48] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:11:48] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:11:48] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:11:48] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:11:49] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:11:56] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [18:12:04] Platonides * Re: [Toolserver-l] Qcronsub and qstat trickery [18:12:07] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:12:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:12:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:12:47] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:12:47] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [18:12:47] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:12:56] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:12:56] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:12:56] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:13:26] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:13:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:13:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:13:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55452.000000 [18:13:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 628410.000000 [18:13:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:13:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:13:56] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:13:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:14:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1065841.000000 [18:14:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:14:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:14:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:14:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35918.000000 [18:14:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:36] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:16:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:16:47] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:16:56] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [18:17:36] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:17:57] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [18:24:04] Platonides * Re: [Toolserver-l] Qcronsub and qstat trickery [18:37:16] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14494.000000 [18:59:06] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3599.000000 [18:59:47] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3526 [19:07:26] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2183 [19:07:56] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2214.000000 [19:09:07] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:09:38] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:09:38] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:09:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 277926.000000 [19:09:38] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62175 MB (10% inode=99%): [19:09:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:09:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:10:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 938927.000000 [19:10:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:06] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:10:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:10:48] / on thyme is UNKNOWN: NRPE: Unable to read output [19:10:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:10:57] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:10:57] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:10:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34519 [19:10:57] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:11:03] [[User talk:Luxo]] ! 10https://wiki.toolserver.org/w/index.php?diff=7918&oldid=578&rcid=21779 * 141.0.10.179 * (-257) (Delete) [19:11:06] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:11:06] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:11:06] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:11:06] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:11:37] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:11:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:11:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:11:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:11:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:11:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 272248 MB (5% inode=62%): [19:11:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:11:56] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:11:57] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:11:57] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:11:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:11:57] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:11:57] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [19:12:02] [[User talk:Luxo]] 10https://wiki.toolserver.org/w/index.php?diff=7919&oldid=7918&rcid=21780 * MZMcBride * (+257) (rv) [19:12:06] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:12:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:12:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:12:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:12:47] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [19:12:52] [[User talk:Luxo]] ! 10https://wiki.toolserver.org/w/index.php?diff=7920&oldid=7919&rcid=21781 * 141.0.10.179 * (-575) (Replaced content with "Delete page") [19:12:56] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:12:56] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:12:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:12:57] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:12:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:13:26] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:13:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:13:28] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:13:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:13:28] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33944.000000 [19:13:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 632010.000000 [19:13:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:13:29] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:13:56] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:13:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:14:08] [[User talk:Luxo]] 10https://wiki.toolserver.org/w/index.php?diff=7921&oldid=7920&rcid=21782 * MZMcBride * (+575) (rv) [19:14:17] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1067860.000000 [19:14:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:14:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:14:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:14:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39518.000000 [19:14:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:14:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:15:34] [[User talk:Luxo]] ! 10https://wiki.toolserver.org/w/index.php?diff=7922&oldid=7921&rcid=21783 * 141.0.10.179 * (-568) (Delete all summary) [19:15:47] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:16:02] [[User talk:Luxo]] 10https://wiki.toolserver.org/w/index.php?diff=7923&oldid=7922&rcid=21784 * MZMcBride * (+568) (rv) [19:16:42] [[User talk:141.0.10.179]] N 10https://wiki.toolserver.org/w/index.php?oldid=7924&rcid=21785 * MZMcBride * (+197) (new page) [19:16:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:16:47] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:16:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [19:17:36] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:17:57] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [19:22:04] [[~luxo/contributions/contributions.php]] ! 10https://wiki.toolserver.org/w/index.php?diff=7925&oldid=7639&rcid=21786 * 141.0.10.179 * (-295) (Delete) [19:23:24] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:141.0.10.17910]] with an expiry time of 1 month (anonymous users only): inappropriate behavior) [19:23:35] [[~luxo/contributions/contributions.php]] 10https://wiki.toolserver.org/w/index.php?diff=7926&oldid=7925&rcid=21788 * MZMcBride * (+295) (rv) [19:23:51] you beat me to the block [19:24:13] I win. [19:26:48] MySQL slave on rosemary is OK: Uptime: 261369 Threads: 35 Questions: 176498888 Slow queries: 107516 Opens: 13933 Flush tables: 1 Open tables: 1379 Queries per second avg: 675.286 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1794 [19:27:06] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1780.000000 [19:31:27] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3623 [19:31:56] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [19:38:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18154.000000 [20:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 281526.000000 [20:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61446 MB (10% inode=99%): [20:09:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:09:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:10:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 942527.000000 [20:10:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:10:06] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:10:47] / on thyme is UNKNOWN: NRPE: Unable to read output [20:10:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:10:56] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:10:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:10:56] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21070 [20:10:56] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:06] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:07] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:11:07] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:07] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:11:37] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:11:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:11:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:11:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 271854 MB (5% inode=62%): [20:11:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:11:57] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:57] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:57] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:57] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:11:57] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [20:12:06] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:12:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:12:17] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:12:47] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [20:12:56] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:12:57] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:12:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:12:57] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:12:57] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:13:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:13:57] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:14:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1070014.000000 [20:14:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:14:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:14:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:14:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:14:28] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:14:28] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19830.000000 [20:14:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 635600.000000 [20:14:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:14:29] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42938.000000 [20:15:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:48] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:16:47] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:16:48] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:16:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [20:17:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:17:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [20:31:27] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7223 [20:31:57] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7254.000000 [20:39:16] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17764.000000 [21:08:47] /tmp on willow is WARNING: DISK WARNING - free space: / 15758 MB (15% inode=98%): [21:08:48] / on willow is WARNING: DISK WARNING - free space: / 15758 MB (15% inode=98%): [21:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:09:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 284482.000000 [21:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61216 MB (10% inode=99%): [21:09:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:09:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:10:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 945927.000000 [21:10:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:10:06] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:11:06] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:11:06] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:11:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:11:07] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:11:07] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:11:37] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:11:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:11:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:11:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:11:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:11:47] / on thyme is UNKNOWN: NRPE: Unable to read output [21:11:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:11:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:11:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 267798 MB (5% inode=62%): [21:11:56] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:11:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:11:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10509 [21:11:57] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:12:06] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:12:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:12:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:12:48] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [21:12:57] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:12:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:12:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:12:57] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:12:57] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:12:57] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:12:57] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:12:58] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [21:12:58] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:12:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:12:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:14:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1068603.000000 [21:14:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:14:57] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:15:16] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:16] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37845.000000 [21:15:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:26] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:26] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:26] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:26] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10388.000000 [21:15:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:15:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 634112.000000 [21:15:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:48] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:16:57] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [21:17:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:17:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:17:47] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [21:18:56] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [21:26:48] /tmp on willow is OK: DISK OK - free space: / 26930 MB (25% inode=99%): [21:26:48] / on willow is OK: DISK OK - free space: / 26930 MB (25% inode=99%): [21:31:26] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10823 [21:32:47] / on wolfsbane is WARNING: DISK WARNING - free space: / 5833 MB (19% inode=93%): [21:32:56] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5832 MB (19% inode=93%): [21:32:56] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10914.000000 [21:39:17] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10010.000000 [22:09:07] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:09:37] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:09:37] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:09:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 287384.000000 [22:09:37] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60940 MB (9% inode=99%): [22:09:47] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:09:47] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:10:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 949247.000000 [22:10:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:06] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:11:06] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:11:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:11:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:11:07] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:11:47] / on thyme is UNKNOWN: NRPE: Unable to read output [22:11:47] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:11:47] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 268604 MB (5% inode=62%): [22:11:47] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:11:56] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:11:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:11:56] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8163 [22:11:57] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:12:06] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:12:06] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:12:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:12:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:12:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:12:37] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:12:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:12:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:12:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:12:37] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:12:47] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [22:13:56] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:13:57] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:13:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:13:57] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:13:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:13:57] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:13:57] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:13:58] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [22:13:58] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:13:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:13:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:14:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1066761.000000 [22:15:16] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:26] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:26] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:26] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:26] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:15:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 630006.000000 [22:15:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8375.000000 [22:15:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:47] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:15:57] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:15:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:17] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35028.000000 [22:16:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:56] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [22:17:36] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:17:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:17:48] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:18:57] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [22:27:56] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 18982 MB (1% inode=99%): [22:28:16] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3503.000000 [22:32:26] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14483 [22:32:47] / on wolfsbane is WARNING: DISK WARNING - free space: / 5703 MB (19% inode=93%): [22:32:57] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5702 MB (19% inode=93%): [22:32:57] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14514.000000 [22:40:16] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1735.000000 [23:07:47] / on wolfsbane is OK: DISK OK - free space: / 6377 MB (21% inode=93%): [23:07:56] /tmp on wolfsbane is OK: DISK OK - free space: / 6378 MB (21% inode=93%): [23:09:06] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:09:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:09:48] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:10:06] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 952608.000000 [23:10:06] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:06] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:36] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:10:36] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 290346.000000 [23:10:36] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60762 MB (9% inode=99%): [23:11:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:11:07] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:11:07] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:11:07] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:11:48] / on thyme is UNKNOWN: NRPE: Unable to read output [23:11:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:11:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 268298 MB (5% inode=62%): [23:11:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:11:57] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:11:57] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:12:06] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:12:06] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:12:06] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:12:36] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:12:36] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:12:37] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:12:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:12:37] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:12:47] APT on yucca is CRITICAL: APT CRITICAL: 6 packages available for upgrade (5 critical updates). [23:12:56] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:12:57] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5720 [23:13:16] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:17] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:13:57] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:13:57] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:13:57] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:13:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:13:57] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:13:57] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:13:57] APT on sage is CRITICAL: APT CRITICAL: 8 packages available for upgrade (5 critical updates). [23:13:58] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:13:58] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:13:59] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:13:59] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:14:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1063744.000000 [23:15:17] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:27] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:27] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:27] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:15:27] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5524.000000 [23:15:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 626708.000000 [23:15:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:28] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:47] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:15:56] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:15:57] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:16] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:17] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:17] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29845.000000 [23:16:17] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:16:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:47] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:17:47] APT on z-dat-s1-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:17:56] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [23:18:37] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:18:57] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [23:32:27] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18083 [23:32:57] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18114.000000 [23:38:27] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3589.000000 [23:38:56] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3545 [23:39:36] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:48:27] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1742.000000 [23:48:57] MySQL slave on z-dat-s5-b is OK: Uptime: 59820 Threads: 4 Questions: 192023281 Slow queries: 434 Opens: 12968 Flush tables: 1 Open tables: 256 Queries per second avg: 3210.18 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1659 [23:56:47] / on wolfsbane is WARNING: DISK WARNING - free space: / 6288 MB (20% inode=93%): [23:56:56] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6282 MB (20% inode=93%):