[00:12:40] nacht ts [00:15:24] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:15:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:15:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:15:26] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:15:36] aliasd on nightshade is CRITICAL: Connection refused [00:15:36] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1587716.000000 [00:15:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:45] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:15:46] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:15:55] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:15:55] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:55] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:15:56] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 276574.000000 [00:15:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 954908.000000 [00:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 764465.000000 [00:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:16:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1168633.000000 [00:16:05] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:05] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 551317.000000 [00:16:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:16:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68432 MB (11% inode=99%): [00:16:15] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:16:15] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:15] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:16:17] / on thyme is UNKNOWN: NRPE: Unable to read output [00:16:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:26] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:27] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [00:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [00:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [00:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:45:24] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324443 MB (6% inode=62%): [00:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21651.000000 [01:00:04] [[Talk:Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=7940&oldid=7938&rcid=21809 * Xaris333 * (+61) (/* CatScan between languages */ ) [01:01:06] [[User talk:Xaris333]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7941&rcid=21810 * Xaris333 * (+46) (Created page with "My page is in [[:el:Χρήστης:Xaris333]].") [01:01:21] [[User:Xaris333]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7942&rcid=21811 * Xaris333 * (+46) (Created page with "My page is in [[:el:Χρήστης:Xaris333]].") [01:02:07] [[User talk:Xaris333]] ! 10https://wiki.toolserver.org/w/index.php?diff=7943&oldid=7941&rcid=21812 * Xaris333 * (+67) () [01:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:15:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:15:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:15:25] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:15:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:15:35] aliasd on nightshade is CRITICAL: Connection refused [01:15:36] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1591056.000000 [01:15:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:15:46] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:55] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:15:55] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:56] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 280174.000000 [01:15:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 955191.000000 [01:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 762915.000000 [01:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:16:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1172232.000000 [01:16:05] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 554917.000000 [01:16:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68202 MB (11% inode=99%): [01:16:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:16:15] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:16:15] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:15] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:25] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:25] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:25] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:17:14] / on thyme is UNKNOWN: NRPE: Unable to read output [01:17:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [01:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [01:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [01:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:25] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324334 MB (6% inode=62%): [01:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25251.000000 [01:59:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67019 MB (10% inode=99%): [02:11:25] Load avg. on ortelius is WARNING: WARNING - load average: 8.18, 18.25, 13.09 [02:13:25] Load avg. on ortelius is OK: OK - load average: 4.01, 13.40, 11.89 [02:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:15:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:15:26] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:15:26] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:36] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:36] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:15:36] aliasd on nightshade is CRITICAL: Connection refused [02:15:36] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:37] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:44] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1594372.000000 [02:15:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:15:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:15:46] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:15:56] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:56] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:56] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:15:56] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:15:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:15:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:15:56] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 283775.000000 [02:15:57] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 954541.000000 [02:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 761309.000000 [02:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:16:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1175833.000000 [02:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 558517.000000 [02:16:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:16:15] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:16:15] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:15] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:25] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:25] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:25] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:17:15] / on thyme is UNKNOWN: NRPE: Unable to read output [02:18:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [02:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [02:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [02:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:25] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324700 MB (6% inode=62%): [02:46:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69246 MB (11% inode=99%): [02:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28851.000000 [03:15:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:15:26] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:15:26] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:15:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:36] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:15:36] aliasd on nightshade is CRITICAL: Connection refused [03:15:36] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:44] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1597691.000000 [03:15:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:15:46] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:55] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:15:56] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:56] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 287374.000000 [03:15:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 953432.000000 [03:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 759975.000000 [03:16:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:16:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1179432.000000 [03:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 562117.000000 [03:16:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:16:15] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:16:15] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:15] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:25] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:25] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:25] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:17:15] / on thyme is UNKNOWN: NRPE: Unable to read output [03:19:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [03:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [03:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [03:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:25] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325109 MB (6% inode=62%): [03:46:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67624 MB (11% inode=99%): [03:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32451.000000 [04:15:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:15:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:15:25] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:15:36] aliasd on nightshade is CRITICAL: Connection refused [04:15:36] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:37] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1601028.000000 [04:15:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:15:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:15:47] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:15:55] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:15:55] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:57] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 290975.000000 [04:16:04] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 953795.000000 [04:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 759289.000000 [04:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:16:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1183032.000000 [04:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:14] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 565717.000000 [04:16:15] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:16:15] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:16] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:16:26] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:26] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:26] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:26] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:17:15] / on thyme is UNKNOWN: NRPE: Unable to read output [04:20:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [04:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [04:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [04:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:45:25] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325276 MB (6% inode=62%): [04:46:14] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67468 MB (11% inode=99%): [04:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36051.000000 [05:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:15:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:15:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:15:25] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:35] aliasd on nightshade is CRITICAL: Connection refused [05:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:15:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1604445.000000 [05:15:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:15:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:56] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:15:56] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:15:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:15:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:15:57] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:16:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 294577.000000 [05:16:06] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 953042.000000 [05:16:06] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 758914.000000 [05:16:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:16] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 569317.000000 [05:16:16] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1186637.000000 [05:16:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:16:16] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:16] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:16] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:16:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:26] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:26] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:16:45] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:17:15] / on thyme is UNKNOWN: NRPE: Unable to read output [05:21:15] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [05:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [05:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [05:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:05] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 103765 MB (10% inode=99%): [05:45:25] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325662 MB (6% inode=62%): [05:46:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67314 MB (11% inode=99%): [05:46:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39651.000000 [06:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:15:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:15:26] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:15:26] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:35] aliasd on nightshade is CRITICAL: Connection refused [06:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:15:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:15:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1607720.000000 [06:15:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:55] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:15:55] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:15:55] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:15:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:15:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:16:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 298177.000000 [06:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 953458.000000 [06:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 759129.000000 [06:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:16:05] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:15] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1190237.000000 [06:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 572917.000000 [06:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:16:15] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:15] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:15] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:25] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:26] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:16:44] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:17:14] / on thyme is UNKNOWN: NRPE: Unable to read output [06:21:15] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [06:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [06:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:24] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325750 MB (6% inode=62%): [06:46:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:46:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 67115 MB (11% inode=99%): [06:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43251.000000 [06:52:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 67088 MB (10% inode=99%): [07:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:15:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:15:26] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:15:26] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:15:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:35] aliasd on nightshade is CRITICAL: Connection refused [07:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:15:36] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:46] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1609689.000000 [07:15:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:15:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:56] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:15:56] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:15:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:16:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 301777.000000 [07:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 954507.000000 [07:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 759040.000000 [07:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 576518.000000 [07:16:15] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1193837.000000 [07:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:16:15] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:16] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:16] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:25] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:26] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:16:45] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:17:15] / on thyme is UNKNOWN: NRPE: Unable to read output [07:21:24] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [07:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [07:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [07:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:45:25] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 326187 MB (6% inode=62%): [07:46:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:50:04] Silke Meyer * [Toolserver-l] Reminder: Toolserver Office Hour April 30th, 4 p.m. UTC [07:52:04] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46851.000000 [07:52:14] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66903 MB (10% inode=99%): [08:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:15:26] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:15:26] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:15:26] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:15:35] aliasd on nightshade is CRITICAL: Connection refused [08:15:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:35] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:15:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1612992.000000 [08:15:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:55] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:15:55] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:15:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:15:55] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:15:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:16:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 305377.000000 [08:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 955367.000000 [08:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 759759.000000 [08:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:16:05] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 580118.000000 [08:16:15] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1197438.000000 [08:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:16:15] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:16] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:16] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:25] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:26] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:16:44] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:17:15] / on thyme is UNKNOWN: NRPE: Unable to read output [08:17:35] Free Memory on damiana is WARNING: WARNING - 6.3% (525976 kB) free! [08:21:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [08:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [08:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [08:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:24:35] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (354524 kB) free! [08:27:35] Free Memory on damiana is OK: OK - 10.5% (878864 kB) free. [08:45:25] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 326119 MB (6% inode=62%): [08:46:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50451.000000 [08:52:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66654 MB (10% inode=99%): [09:07:35] Free Memory on damiana is WARNING: WARNING - 5.6% (472860 kB) free! [09:10:36] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (419120 kB) free! [09:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:36] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:15:36] aliasd on nightshade is CRITICAL: Connection refused [09:15:36] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:37] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:15:46] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:46] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:15:46] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1616281.000000 [09:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:15:46] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:15:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:56] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:56] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:15:56] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:15:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:15:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:15:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:16:04] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 308976.000000 [09:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 954907.000000 [09:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 760378.000000 [09:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:16:05] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 583717.000000 [09:16:15] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1201038.000000 [09:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:16:15] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:15] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:16] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:16:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:16:26] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:26] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:26] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:16:26] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:16:26] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:26] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:26] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:27] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:27] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:16:45] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:17:15] / on thyme is UNKNOWN: NRPE: Unable to read output [09:21:29] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [09:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [09:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [09:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:25] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321204 MB (6% inode=62%): [09:46:16] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:47:05] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 4.918 second response time [09:47:55] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.418 second response time [09:49:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318924 MB (5% inode=62%): [09:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54051.000000 [09:52:15] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66452 MB (10% inode=99%): [09:58:25] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324263 MB (6% inode=62%): [10:10:35] Free Memory on damiana is CRITICAL: CRITICAL - 2.5% (208564 kB) free! [10:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:35] aliasd on nightshade is CRITICAL: Connection refused [10:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:15:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1619560.000000 [10:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:56] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:15:56] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:15:57] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 954576.000000 [10:16:05] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 312577.000000 [10:16:06] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:16:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 760579.000000 [10:16:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:16:14] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1204638.000000 [10:16:14] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 587318.000000 [10:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:16:16] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:16:16] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:16:16] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:16:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:16:26] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:16:26] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:16:26] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:16:45] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:16:45] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:16:45] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:17:15] / on thyme is UNKNOWN: NRPE: Unable to read output [10:17:25] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:17:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:17:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:17:26] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:17:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:17:26] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:21:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:21:34] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [10:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [10:21:36] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [10:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:38:45] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:38:55] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:39:25] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:40:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:40:06] Here we go again [10:40:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:40:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:41:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:41:59] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:59] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 587064.000000 [10:42:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:42:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:42:03] / on thyme is UNKNOWN: NRPE: Unable to read output [10:42:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:13] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:13] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:23] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:52:53] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57699.000000 [10:53:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66317 MB (10% inode=99%): [10:58:53] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.419 second response time [10:59:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324470 MB (6% inode=62%): [11:02:43] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.583 second response time [11:05:53] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.826 second response time [11:11:23] Free Memory on damiana is CRITICAL: CRITICAL - 1.8% (147200 kB) free! [11:12:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:41:13] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:41:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:41:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:41:14] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:14] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:14] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:15] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [11:41:23] aliasd on nightshade is CRITICAL: Connection refused [11:41:24] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:41:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:24] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [11:41:24] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [11:41:24] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:24] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:32] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:33] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:41:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:41:43] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:43] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:41:44] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:41:44] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:41:44] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:41:44] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 317722.000000 [11:41:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 955007.000000 [11:41:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 761196.000000 [11:41:53] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:41:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1199029.000000 [11:41:54] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:54] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 585459.000000 [11:42:03] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:42:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:03] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:42:04] / on thyme is UNKNOWN: NRPE: Unable to read output [11:42:05] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:42:13] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:13] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:13] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:13] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:23] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1624100.000000 [11:42:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:33] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:42:33] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:42:33] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:42:34] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:52:53] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61299.000000 [11:53:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66094 MB (10% inode=99%): [11:59:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324325 MB (6% inode=62%): [12:11:24] Free Memory on damiana is CRITICAL: CRITICAL - 1.5% (127724 kB) free! [12:12:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:41:13] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:41:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:41:14] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:41:14] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:14] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:15] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:15] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [12:41:23] aliasd on nightshade is CRITICAL: Connection refused [12:41:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:41:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:23] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [12:41:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [12:41:24] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:24] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:33] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:41:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:41:43] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:43] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:43] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:43] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:43] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:43] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:44] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:44] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 321323.000000 [12:41:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 955505.000000 [12:41:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:53] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:54] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 761354.000000 [12:41:54] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:41:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1195568.000000 [12:41:54] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:54] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 583817.000000 [12:42:03] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:42:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:42:03] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:03] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:03] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:04] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:42:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:42:05] / on thyme is UNKNOWN: NRPE: Unable to read output [12:42:13] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:42:13] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:13] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:14] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:14] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:14] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:14] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:23] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:42:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1627419.000000 [12:42:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:42:34] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:42:34] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:42:34] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:42:34] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:52:53] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64899.000000 [12:53:03] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65857 MB (10% inode=99%): [12:59:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324222 MB (6% inode=62%): [13:06:04] And down we go again [13:13:04] down again [13:13:06] :@ [13:20:45] sigh [13:28:48] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:28:49] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:28:49] Sun Grid Engine execd on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:31:06] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:31:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:31:06] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:33:24] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [13:33:24] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [13:33:34] Free Memory on damiana is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:34:34] ts-array5 on damiana is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:34:34] Environment IPMI on damiana is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:34:34] DiskSuite on damiana is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:35:06] Load avg. on nightshade is WARNING: WARNING - load average: 8.77, 15.41, 18.76 [13:35:14] SSH on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:14] HTTP proxy on ha-proxy.esi is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:35:14] Sun Grid Engine execd on mayapple is UNKNOWN: Execution timeout exceeded [13:36:14] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:05] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2187.000000 [13:37:05] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2197.000000 [13:37:05] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2216.000000 [13:37:06] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2200.000000 [13:37:06] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2210.000000 [13:38:19] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [13:38:19] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [13:38:19] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [13:38:20] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [13:38:28] Sun Grid Engine execd on mayapple is CRITICAL: CRITICAL: execd not communicating [13:38:28] Sun Grid Engine execd on yarrow is CRITICAL: CRITICAL: execd not communicating [13:38:28] Sun Grid Engine execd on nightshade is CRITICAL: CRITICAL: execd not communicating [13:39:19] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2116 [13:39:20] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2261 [13:39:20] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2328 [13:39:20] HTTP proxy on ha-proxy.esi is OK: HTTP OK: HTTP/1.0 302 Moved Temporarily - 1114 bytes in 0.053 second response time [13:39:20] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2312 [13:39:20] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [13:39:20] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [13:40:19] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1650.000000 [13:40:19] MySQL slave on daphne is OK: Uptime: 1014818 Threads: 24 Questions: 114327204 Slow queries: 226072 Opens: 45544 Flush tables: 1 Open tables: 1695 Queries per second avg: 112.657 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1678 [13:40:20] MySQL slave on cassia is OK: Uptime: 1018501 Threads: 10 Questions: 980981503 Slow queries: 14041 Opens: 14219 Flush tables: 1 Open tables: 13973 Queries per second avg: 963.162 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1631 [13:40:20] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1645.000000 [13:40:20] Sun Grid Engine execd on mayapple is OK: Host and Queues Ok [13:40:20] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.778 second response time [13:41:19] SMTP on damiana is OK: SMTP OK - 0.479 sec. response time [13:41:20] SSH on damiana is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [13:41:20] MySQL slave on z-dat-s7-a is OK: Uptime: 490236 Threads: 6 Questions: 401071859 Slow queries: 7227 Opens: 630970 Flush tables: 1 Open tables: 3425 Queries per second avg: 818.119 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1755 [13:41:28] ts-array5 on damiana is OK: 2/2 paths are active [13:41:28] Free Memory on damiana is OK: OK - 85.6% (7174452 kB) free. [13:41:28] Environment IPMI on damiana is OK: ok: temperature ok fan ok voltage ok chassis ok [13:42:30] Load avg. on nightshade is OK: OK - load average: 7.00, 9.20, 14.45 [13:46:24] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.563 second response time [13:47:21] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.282 second response time [13:51:21] MySQL slave on rosemary is OK: Uptime: 321523 Threads: 18 Questions: 216945850 Slow queries: 59243 Opens: 20654 Flush tables: 1 Open tables: 3180 Queries per second avg: 674.744 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1736 [13:51:39] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1676.000000 [14:00:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3619.000000 [14:00:39] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000 [14:02:40] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:02:40] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:02:48] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:02:49] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 765134.000000 [14:03:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:03:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 326211.000000 [14:03:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 960126.000000 [14:03:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1198762.000000 [14:03:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:03:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:03:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:03:22] aliasd on nightshade is CRITICAL: Connection refused [14:03:23] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [14:03:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:03:24] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [14:03:24] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [14:03:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:03:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:29] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:29] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:39] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:39] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:39] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:48] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:48] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:03:48] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:48] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:48] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:49] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:49] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:49] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:50] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:50] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:51] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:51] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:52] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:03:52] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:04:08] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:04:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:04:22] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 587390.000000 [14:04:22] / on thyme is UNKNOWN: NRPE: Unable to read output [14:04:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:04:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1632116.000000 [14:04:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:04:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:04:29] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:04:29] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:04:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:04:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:04:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:04:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:14:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69784.000000 [14:15:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65623 MB (10% inode=99%): [14:21:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324502 MB (6% inode=62%): [14:31:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:31:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:31:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:01:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7278.000000 [15:01:39] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7285.000000 [15:02:40] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:02:40] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:02:49] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:02:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:03:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 768734.000000 [15:03:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 329810.000000 [15:03:21] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 963726.000000 [15:03:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1202362.000000 [15:03:39] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:48] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:03:49] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:49] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:49] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:49] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:49] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:50] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:50] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:51] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:51] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:52] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:52] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:03:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:04:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:04:08] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:04:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 590990.000000 [15:04:21] / on thyme is UNKNOWN: NRPE: Unable to read output [15:04:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:04:21] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:04:21] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:04:21] aliasd on nightshade is CRITICAL: Connection refused [15:04:21] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [15:04:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:04:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [15:04:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [15:04:28] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:04:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:04:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:04:29] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:04:29] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:04:39] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:04:39] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:04:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:04:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:49] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:04:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:05:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:05:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1635776.000000 [15:05:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:05:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:14:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73384.000000 [15:15:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65355 MB (10% inode=99%): [15:18:53] *Argl*, aliasd is stuck again on yarrow. [15:21:20] And nightshade. [15:21:21] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325227 MB (6% inode=62%): [15:22:01] Platonides: Do you know whether "operators" can restart services? [15:31:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:31:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:31:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:33:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:40:20] @replag [15:40:21] Betacommand: s1-rr-a: 20h 49m 16s [-]; s1-rr-a-wd: 1w 2d 9h 14m 58s [-]; s1-user-c: 3d 20h 14m 2s [-]; s1-user-wd: 1w 6d 22h 36m 32s [-]; s2-rr: 20h 49m 35s [-]; s2-user: 20h 49m 35s [-]; s2-user-c: error; s2-user-wd: 1w 4d 4h 19m 17s [-] [15:40:22] Betacommand: s3-user: 11s [-]; s3-user-wd: 1w 2d 16h 41m 59s [-]; s4-user-wd: 2w 4d 22h 58m 2s [-]; s5-rr-a-wd: 6d 20h 46m 8s [-]; s5-user: 20h 49m 20s [-]; s5-user-c: 2h 40m 17s [-]; s6-user-wd: 1w 3d 1h 35m 21s [-]; s7-user-wd: 1w 1d 22h 9m 34s [-] [15:56:21] office hour in 5 minutes with Silke_WMDE_ Coren and me in #wikimedia-office [16:01:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10879.000000 [16:01:39] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10885.000000 [16:02:48] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:02:49] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:03:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:03:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 772334.000000 [16:03:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 333410.000000 [16:03:21] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 967326.000000 [16:03:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1205961.000000 [16:03:39] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:03:39] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:03:39] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:03:48] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:03:48] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:03:48] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:03:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:04:07] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:04:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:04:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 594590.000000 [16:04:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:04:22] / on thyme is UNKNOWN: NRPE: Unable to read output [16:04:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:04:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:04:22] aliasd on nightshade is CRITICAL: Connection refused [16:04:22] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [16:04:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:04:23] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [16:04:24] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [16:04:29] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:40] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:40] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:48] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:48] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:48] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:49] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:49] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:49] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:49] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:50] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:50] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:04:51] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:05:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:05:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1639376.000000 [16:05:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:05:21] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:05:48] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:05:48] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:48] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:04] Tim Landscheidt * Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [16:14:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76984.000000 [16:15:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65069 MB (10% inode=99%): [16:21:21] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324962 MB (6% inode=63%): [16:31:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:31:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:31:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:33:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:01:39] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14486.000000 [17:01:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14480.000000 [17:02:49] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:02:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:03:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:03:22] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 775934.000000 [17:03:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 337010.000000 [17:03:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 970927.000000 [17:03:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1209562.000000 [17:03:39] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:03:39] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:03:39] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:03:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:03:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:49] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:58] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:04:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:04:08] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:04:22] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 598189.000000 [17:04:22] / on thyme is UNKNOWN: NRPE: Unable to read output [17:04:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:04:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:04:23] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:04:23] aliasd on nightshade is CRITICAL: Connection refused [17:04:23] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [17:04:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:04:24] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [17:04:24] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [17:04:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:29] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:30] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:38] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:39] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:39] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:48] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:49] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:04:49] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:49] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:49] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:49] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:50] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:50] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:51] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:51] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:04:52] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:05:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:05:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1642976.000000 [17:05:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:05:29] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:05:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:05:49] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:05:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80584.000000 [17:15:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64809 MB (10% inode=99%): [17:21:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325048 MB (6% inode=63%): [17:31:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:31:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:31:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:33:18] @replag [17:33:19] Betacommand: s1-rr-a: 22h 42m 14s [+1.00 s/s]; s1-rr-a-wd: 1w 2d 11h 7m 56s [+1.00 s/s]; s1-user-c: 3d 22h 7m 0s [+1.00 s/s]; s1-user-wd: 2w 29m 30s [+1.00 s/s]; s2-rr: 22h 42m 33s [+1.00 s/s]; s2-user: 22h 42m 33s [+1.00 s/s]; s2-user-c: error; s2-user-wd: 1w 4d 6h 12m 15s [+1.00 s/s] [17:33:20] Betacommand: s3-user-wd: 1w 2d 18h 34m 57s [+1.00 s/s]; s4-user-wd: 2w 5d 50m 59s [+1.00 s/s]; s5-rr-a-wd: 6d 22h 39m 6s [+1.00 s/s]; s5-user: 22h 42m 18s [+1.00 s/s]; s5-user-c: 4h 33m 15s [+1.00 s/s]; s6-user-wd: 1w 3d 3h 28m 19s [+1.00 s/s]; s7-user-wd: 1w 2d 2m 32s [+1.00 s/s] [17:33:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:01:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18080.000000 [18:01:39] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18087.000000 [18:02:21] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 103442 MB (10% inode=99%): [18:02:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:02:49] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:03:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:03:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 779534.000000 [18:03:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 340610.000000 [18:03:22] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 974527.000000 [18:03:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1213162.000000 [18:03:39] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:03:40] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:03:40] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:03:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:04:07] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:04:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:04:39] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:39] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:40] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:40] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:40] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:04:49] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:49] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:04:49] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:49] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:50] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:50] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:51] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:51] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:52] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:52] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:04:53] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:05:20] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 601849.000000 [18:05:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:05:21] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:05:21] / on thyme is UNKNOWN: NRPE: Unable to read output [18:05:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:05:21] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:05:22] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [18:05:22] aliasd on nightshade is CRITICAL: Connection refused [18:05:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [18:05:23] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [18:05:24] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:05:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1646576.000000 [18:05:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:05:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:05:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:48] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:05:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:49] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:14:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84184.000000 [18:15:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64572 MB (10% inode=99%): [18:21:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325018 MB (6% inode=63%): [18:31:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:31:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:31:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:33:31] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:01:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21681.000000 [19:01:40] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21687.000000 [19:01:59] hello all [19:02:49] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:02:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:03:20] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:03:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 783135.000000 [19:03:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 344210.000000 [19:03:39] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:03:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:03:40] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:03:58] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:04:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:04:08] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:04:21] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 978187.000000 [19:04:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1216822.000000 [19:04:39] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:39] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:40] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:48] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:04:48] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:49] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:49] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:49] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:49] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:50] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:50] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:51] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:51] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:52] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:52] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:04:53] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:05:22] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 605450.000000 [19:05:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:05:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:05:22] / on thyme is UNKNOWN: NRPE: Unable to read output [19:05:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:05:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:05:22] aliasd on nightshade is CRITICAL: Connection refused [19:05:23] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [19:05:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:05:24] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [19:05:24] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [19:05:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1650176.000000 [19:05:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:05:29] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:05:29] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:05:39] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:05:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:05:39] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:05:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:06:48] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:06:49] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:06:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:42] DaBPunkt: Hi! Could you restart aliasd on nightshade and yarrow, please? Thanks. [19:14:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 87784.000000 [19:15:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64333 MB (10% inode=99%): [19:16:17] scfc_de: sure. wait a moment [19:17:23] done [19:19:24] DaBPunkt: Thanks. [19:21:21] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324923 MB (6% inode=63%): [19:31:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:31:39] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:32:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:33:39] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:59:21] Free Memory on turnera is WARNING: WARNING - 5.1% (424416 kB) free! [20:01:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25281.000000 [20:02:39] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25347.000000 [20:02:49] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:02:49] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:39] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:39] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:48] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:03:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:04:08] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:04:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:04:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:04:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 786794.000000 [20:04:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 347869.000000 [20:04:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1220422.000000 [20:04:21] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 981787.000000 [20:04:40] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:04:40] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:04:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:04:49] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:04:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:04:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:04:49] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:04:49] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:04:50] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:04:50] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:04:51] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:04:51] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:04:52] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:05:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:05:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:40] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:05:40] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:05:40] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:05:40] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:05:40] aliasd on yarrow is CRITICAL: Connection refused [20:05:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:06:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 609109.000000 [20:06:21] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:06:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:06:21] / on thyme is UNKNOWN: NRPE: Unable to read output [20:06:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:06:21] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:06:21] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [20:06:22] aliasd on nightshade is CRITICAL: Connection refused [20:06:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:06:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [20:06:23] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [20:06:24] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1653836.000000 [20:06:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:06:49] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:06:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:49] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:21] Free Memory on turnera is OK: OK - 7.5% (625684 kB) free. [20:14:20] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91384.000000 [20:15:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64092 MB (10% inode=99%): [20:19:21] Free Memory on turnera is WARNING: WARNING - 5.6% (469700 kB) free! [20:21:21] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 325063 MB (6% inode=63%): [20:31:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:32:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:32:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:34:04] Kolossos * Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [20:46:28] DaBPunkt: The queues on nightshade and yarrow seem to be somewhat stuck again. Could you "postfix flush" on both of them? [20:55:12] DaBPunkt: http seems down: http://toolserver.org/~alchimista/ [20:55:39] works here [20:56:41] DaBPunkt: do we have an eta on getting s1-user fixed? [20:56:45] Hi DaBPunkt, you missed the office hour [20:58:43] DaBPunkt: it's loading very veru slow, it's getting some timeouts [20:58:48] *very [20:59:08] multichill: I have better things to do than reading why-labs-is-so-great ;) [20:59:10] Ahoy - I'm seeing some misbehaviour on http://toolserver.org/~mwbot/botbrain.html [20:59:20] Free Memory on turnera is CRITICAL: CRITICAL - 3.0% (249320 kB) free! [20:59:38] Namely it seems like it's timing out [21:00:05] works here too [21:00:26] even if I manully select wolsfbane or ortelius [21:00:41] it's a cabal.. ts pages are only shown to dab XD [21:00:43] Betacommand: I hope to fix it during May [21:01:14] Alchimista: I have a private lan-cable to the cluster, but psst! ;) [21:01:39] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28881.000000 [21:01:46] :O a fiber cable? [21:02:39] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28947.000000 [21:02:44] DaBPunkt: Wasn't really like that. One thing that really sucks: No database servers with wikipedia + commons. They want to use federation [21:02:53] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:02:53] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:03:06] Anyway, the full log is at http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20130430.txt [21:03:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:03:39] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:03:40] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:03:49] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:03:58] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:04:03] multichill: they shoudl do as they please – I have a working cluster with commons+wikipedia here :-) [21:04:08] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:04:08] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:04:12] but thanks for the log [21:04:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:04:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 351470.000000 [21:04:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 790394.000000 [21:04:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1224022.000000 [21:04:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 985397.000000 [21:04:50] What's wrong with the switch btw? It has been unreachable for quite some time now [21:05:05] Marlen Caemmerer * [Toolserver-announce] Maintenance of s3, s6 and s4 [21:05:27] multichill: nosy told me that it's SNMP doesn't like the new IP-range [21:05:29] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:05:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:39] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:40] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:40] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:45] You probably have to change some hidden acl [21:05:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:49] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:49] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:05:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:49] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:49] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:50] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:50] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:51] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:51] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:52] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:52] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:05:53] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:13] DaBPunkt: The queues on nightshade and yarrow seem to be somewhat stuck again. Could you "postfix flush" on both of them? [21:06:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 612710.000000 [21:06:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:06:21] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:06:21] / on thyme is UNKNOWN: NRPE: Unable to read output [21:06:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:06:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:06:22] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [21:06:23] aliasd on nightshade is CRITICAL: Connection refused [21:06:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:06:24] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [21:06:24] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [21:06:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1657436.000000 [21:06:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:06:39] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:39] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:06:40] aliasd on yarrow is CRITICAL: Connection refused [21:06:48] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:06:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:49] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:00] scfc_de: let me see [21:07:39] aliasd on yarrow is OK: TCP OK - 0.002 second response time on port 984 [500 Not found.] [21:08:23] aliasd on nightshade is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [21:09:45] [[Special:Log/newusers]] create 10 * Sonphan The Cool * (New user account) [21:13:55] [[User talk:Sonphan The Cool]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7944&rcid=21814 * Sonphan The Cool * (+94) (Created page with "Hello now, my name is Son Phan. I was born on September 13, 2003 and I'll show you some techs.") [21:14:27] I have a bad feeling with that user-name… [21:14:52] [[Special:Log/move]] move 10 * Sonphan The Cool * (moved [[02User talk:Sonphan The Cool10]] to [[122 St.]]: I think it should be a user.) [21:15:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 95044.000000 [21:15:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63827 MB (10% inode=99%): [21:15:25] [[Talk:122 St.]] !NM 10https://wiki.toolserver.org/w/index.php?oldid=7947&rcid=21816 * Sonphan The Cool * (+23) (Created page with "The cool is in Main St.") [21:18:00] DaBPunkt: will you give a cool block to Sonphan the cool? Not that cool -> https://wiki.toolserver.org/view/122_St. [21:22:21] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324685 MB (6% inode=63%): [21:31:38] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:32:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:33:29] Dispenser hi htere, I just wrote about a safari bug re: wikimapia links on your en wiki talk page. [21:33:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:34:20] ok [21:34:27] Dispenser thank you. [21:35:38] I suspect the problem is with wikimapia [21:40:13] [[Special:Log/delete]] delete 10 * Gifti * (deleted "[[02User talk:Sonphan The Cool10]]": Spam: content was: "#REDIRECT [[122 St.]]" (and the only contributor was "[[Special:Contributions/Sonphan The Cool|Sonphan The Cool]]")) [21:40:29] [[Special:Log/delete]] delete 10 * Gifti * (deleted "[[02122 St.10]]": Spam: content was: "Hello now, my name is Son Phan. I was born on September 13, 2003 and I'll show you some techs." (and the only contributor was "[[Special:Contributions/Sonphan The Cool|Sonphan The Cool]]")) [21:40:41] [[Special:Log/delete]] delete 10 * Gifti * (deleted "[[02Talk:122 St.10]]": Spam: content was: "The cool is in Main St." (and the only contributor was "[[Special:Contributions/Sonphan The Cool|Sonphan The Cool]]")) [21:50:40] "iPad; CPU OS 6_0 like Mac OS X" tsk tsk tsk Apple [21:59:20] Free Memory on turnera is CRITICAL: CRITICAL - 2.7% (227216 kB) free! [22:01:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23251.000000 [22:02:40] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13816.000000 [22:02:58] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:02:58] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:03:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:03:49] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:03:49] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:03:50] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:03:58] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:04:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:04:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 792153.000000 [22:04:21] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 355071.000000 [22:04:21] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:04:21] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:04:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1224617.000000 [22:04:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 986901.000000 [22:04:45] nacht ts [22:05:39] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:39] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:39] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:40] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:49] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:05:49] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:49] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:49] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:49] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:05:49] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:49] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:50] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:58] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:58] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:58] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:58] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:58] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:58] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:58] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:59] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:05:59] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:06:20] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:06:48] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:06:49] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:50] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 613667.000000 [22:07:21] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:07:21] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:07:21] / on thyme is UNKNOWN: NRPE: Unable to read output [22:07:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:07:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:07:22] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [22:07:22] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:07:22] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [22:07:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [22:07:23] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1660942.000000 [22:07:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:07:39] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:39] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:07:39] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63542 MB (10% inode=99%): [22:16:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 98705.000000 [22:23:20] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 324282 MB (6% inode=63%): [22:25:39] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3250.000000 [22:27:38] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1604.000000 [22:28:21] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6217 MB (20% inode=93%): [22:28:21] / on wolfsbane is WARNING: DISK WARNING - free space: / 6217 MB (20% inode=93%): [22:31:40] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:33:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:34:28] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:39:11] Free Memory on turnera is OK: OK - 91.2% (7640812 kB) free. [22:48:41] [[Special:Log/block]] block 10 * Legoktm * (blocked [[02User:Sonphan The Cool10]] with an expiry time of infinite (account creation disabled): Cross-wiki abuse) [23:01:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9202.000000 [23:03:02] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:03:02] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:03:51] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:03:51] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:04:34] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 988416.000000 [23:04:34] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:04:34] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:04:42] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:05:12] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:05:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 794117.000000 [23:05:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 358731.000000 [23:05:12] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:05:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1225507.000000 [23:05:41] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:01] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:02] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:12] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:12] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:12] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:23] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:23] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:23] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:06:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:42] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:42] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:06:42] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:07:02] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:07:12] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:07:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:07:22] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:07:22] / on thyme is UNKNOWN: NRPE: Unable to read output [23:07:22] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:07:22] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:07:23] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:07:23] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [23:07:24] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [23:07:24] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [23:07:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1664353.000000 [23:07:25] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:07:26] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:08:12] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 615110.000000 [23:11:52] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2188.000000 [23:17:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 63215 MB (10% inode=99%): [23:17:12] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102365.000000 [23:23:22] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 322658 MB (6% inode=63%): [23:28:22] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6091 MB (20% inode=93%): [23:28:22] / on wolfsbane is WARNING: DISK WARNING - free space: / 6091 MB (20% inode=93%): [23:32:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:33:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:33:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:34:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:35:51] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3628.000000 [23:36:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:36:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:36:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:36:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:36:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt.