[00:00:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:00:28] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:00:48] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:00:48] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1157287.000000 [00:01:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:01:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 897442.000000 [00:01:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:01:49] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 558154.000000 [00:01:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:01:58] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 22417 MB (3% inode=99%): [00:01:59] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 189330.000000 [00:01:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:02:08] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:09] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:09] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:09] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:09] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:10] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:10] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:11] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:12] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:12] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:02:28] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:02:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:02:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:02:48] / on thyme is UNKNOWN: NRPE: Unable to read output [00:02:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:02:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:02:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:02:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1504826.000000 [00:03:08] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:03:08] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:03:08] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:03:08] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:03:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:03:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:09] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:03:37] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:03:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:04:08] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:04:08] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:04:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:05:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:08] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:05:08] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:05:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:05:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:08] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:58] / on wolfsbane is WARNING: DISK WARNING - free space: / 6163 MB (20% inode=93%): [00:27:58] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6163 MB (20% inode=93%): [00:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:58:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315600 MB (5% inode=61%): [00:59:48] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 738438.000000 [01:00:09] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:00:27] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:00:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:00:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1153472.000000 [01:01:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:01:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 900350.000000 [01:01:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:01:48] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 557289.000000 [01:01:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:01:58] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 22209 MB (3% inode=99%): [01:01:58] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 192930.000000 [01:01:58] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:02:08] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:02:09] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:02:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:02:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:02:09] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:02:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:02:09] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:02:28] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:02:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:02:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:02:48] / on thyme is UNKNOWN: NRPE: Unable to read output [01:02:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:02:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:02:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:02:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1508082.000000 [01:03:08] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:03:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:08] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:08] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:08] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:08] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:09] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:10] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:11] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:12] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:12] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:03:37] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:03:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:04:07] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:04:08] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:04:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:05:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:08] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:05:08] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:05:08] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:05:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:06:08] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:24:53] Does anyone know why the heck the replag for replicating Wikidata edits is so long? [01:27:58] / on wolfsbane is WARNING: DISK WARNING - free space: / 5020 MB (16% inode=93%): [01:27:59] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 5018 MB (16% inode=93%): [01:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:58:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315514 MB (5% inode=61%): [01:59:48] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 738778.000000 [02:00:07] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:00:28] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:00:48] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:00:48] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1150649.000000 [02:01:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:01:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 903244.000000 [02:01:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:01:49] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556918.000000 [02:01:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:01:58] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20112 MB (3% inode=99%): [02:01:59] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 196531.000000 [02:01:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:02:08] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:02:08] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:02:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:02:09] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:02:09] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:02:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:02:09] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:02:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:02:28] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:02:48] / on thyme is UNKNOWN: NRPE: Unable to read output [02:02:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:02:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:02:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:02:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:02:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1511460.000000 [02:03:08] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:03:08] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:08] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:08] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:09] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:10] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:11] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:11] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:12] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:12] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:13] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:03:37] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:03:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:03:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:04:08] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:04:08] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:04:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:05:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:09] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:05:09] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:05:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:05:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:08] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:13:48] / on ortelius is WARNING: DISK WARNING - free space: / 6262 MB (20% inode=92%): [02:27:58] / on wolfsbane is WARNING: DISK WARNING - free space: / 3782 MB (12% inode=93%): [02:27:58] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3781 MB (12% inode=93%): [02:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:58:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315386 MB (5% inode=61%): [02:59:48] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 738644.000000 [03:00:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:00:28] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:00:48] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:00:48] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1149688.000000 [03:01:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:01:27] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 906136.000000 [03:01:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:01:49] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 556535.000000 [03:01:49] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:01:58] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 22041 MB (3% inode=99%): [03:01:58] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 200131.000000 [03:01:58] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:02:08] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:02:08] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:02:08] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:02:08] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:02:08] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:02:08] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:02:09] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:02:28] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:02:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:02:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:02:48] / on thyme is UNKNOWN: NRPE: Unable to read output [03:02:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:02:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:02:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1514880.000000 [03:02:59] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:03:08] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:03:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:03:08] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:03:08] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:03:08] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:03:09] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:03:09] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:03:10] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:03:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:03:37] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:03:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:03:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:04:08] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:04:08] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:04:08] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:04:08] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:04:08] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:04:08] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:04:08] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:04:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:04:09] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:04:10] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:05:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:05:09] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:05:09] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:05:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:05:09] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:08] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:11:48] / on ortelius is OK: DISK OK - free space: / 8327 MB (27% inode=92%): [03:27:58] / on wolfsbane is WARNING: DISK WARNING - free space: / 3870 MB (12% inode=93%): [03:27:59] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3870 MB (12% inode=93%): [03:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:58:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315238 MB (5% inode=61%): [03:58:58] Free Memory on damiana is WARNING: WARNING - 6.2% (520440 kB) free! [03:59:48] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 739333.000000 [04:00:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:00:28] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:00:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:00:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1148696.000000 [04:01:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:01:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 909092.000000 [04:01:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:01:48] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 557088.000000 [04:01:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:01:58] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21936 MB (3% inode=99%): [04:01:58] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 203732.000000 [04:01:58] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:02:08] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:02:09] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:02:09] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:02:09] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:02:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:02:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:02:09] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:02:27] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:02:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:02:48] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:02:48] / on thyme is UNKNOWN: NRPE: Unable to read output [04:02:48] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:02:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:02:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1518224.000000 [04:02:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:03:08] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:03:08] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:03:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:03:08] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:03:08] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:03:09] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:03:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:03:09] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:03:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:03:37] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:03:48] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:03:49] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:04:07] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:04:09] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:04:09] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:04:09] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:04:09] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:04:09] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:04:09] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:04:09] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:04:10] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:04:10] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:05:08] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:08] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:05:08] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:05:08] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:09] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:06:08] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:11:58] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (377520 kB) free! [04:14:58] Free Memory on damiana is OK: OK - 7.1% (591092 kB) free. [04:15:58] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3277 MB (10% inode=93%): [04:15:59] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3277 MB (10% inode=93%): [04:26:59] Free Memory on damiana is WARNING: WARNING - 6.6% (550668 kB) free! [04:29:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:58:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 315097 MB (5% inode=61%): [04:59:48] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 740322.000000 [05:00:08] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:00:27] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:00:49] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:00:49] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1149267.000000 [05:01:08] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:01:28] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 912001.000000 [05:01:48] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:01:48] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 557249.000000 [05:01:48] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:01:58] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21846 MB (3% inode=99%): [05:01:59] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 207332.000000 [05:01:59] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:02:08] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:02:08] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:02:09] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:02:09] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:02:09] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:02:09] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:02:09] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:02:28] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:02:28] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:02:48] / on thyme is UNKNOWN: NRPE: Unable to read output [05:02:49] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:02:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:02:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:02:58] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1521597.000000 [05:03:00] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:03:08] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:03:08] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:08] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:08] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:09] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:03:09] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:03:09] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:03:09] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:03:10] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:03:37] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:03:49] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:04:19] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:04:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:04:26] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:04:27] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:04:27] FC 0/18 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:04:27] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:04:27] FC 0/23 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:04:28] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:04:28] FC 0/13 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:04:28] FC 0/16 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:04:28] FC 0/20 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [05:04:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:04:29] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:04:48] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: No answer from host [05:04:48] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: No answer from host [05:06:30] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:06:30] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:06:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:30] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:30] FC 0/9 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:31] FC 0/22 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:32] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:33] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:06:34] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [05:40:13] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:40:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:40:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:40:14] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21853 MB (3% inode=99%): [05:40:14] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:14] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:14] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:40:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_SAS_PORT_DEGRADED.description:S27:Tray.8 [05:40:57] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:57] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:57] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:57] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:57] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:57] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:32] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 559721.000000 [05:45:32] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2423.000000 [05:46:12] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [05:46:12] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [05:46:12] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1840.000000 [05:46:12] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2470.000000 [05:46:12] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2407.000000 [05:46:18] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:46:19] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1821 [05:46:31] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:46:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:46:31] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:46:31] / on thyme is UNKNOWN: NRPE: Unable to read output [05:46:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:47:13] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:47:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 914699.000000 [05:47:13] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [05:47:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:47:13] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2494 MB (8% inode=93%): [05:47:13] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2494 MB (8% inode=93%): [05:47:13] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1714.000000 [05:47:19] MySQL slave on rosemary is OK: Uptime: 206084 Threads: 11 Questions: 159559986 Slow queries: 29156 Opens: 13026 Flush tables: 1 Open tables: 3142 Queries per second avg: 774.247 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1706 [05:57:47] SSH on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:57:57] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [05:59:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 314975 MB (5% inode=61%): [06:00:19] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:00:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 743724.000000 [06:00:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:05:11] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [06:05:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3625.000000 [06:06:11] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [06:08:11] aliasd on nightshade is CRITICAL: Connection refused [06:08:20] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:08:40] Sun Grid Engine execd on mayapple is UNKNOWN: Execution timeout exceeded [06:12:38] SSH on nightshade is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [06:12:38] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [06:39:18] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:39:47] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:40:11] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1527304.000000 [06:40:11] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1155019.000000 [06:40:11] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:40:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 213224.000000 [06:40:19] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:40:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [06:40:32] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:32] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:32] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:32] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:40:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:33] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:34] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:34] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:35] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:35] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:36] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:48] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:40:48] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:41:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21700 MB (3% inode=99%): [06:41:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:41:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:41:38] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:41:38] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:41:38] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:41:38] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:41:38] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:32] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 563324.000000 [06:46:31] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:46:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:46:31] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:46:32] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:46:32] / on thyme is UNKNOWN: NRPE: Unable to read output [06:46:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:47:11] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [06:47:11] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [06:48:10] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:48:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 918359.000000 [06:48:11] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [06:48:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:48:12] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 1700 MB (5% inode=93%): [06:48:12] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 1700 MB (5% inode=93%): [06:59:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 314644 MB (5% inode=61%): [07:00:18] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:00:31] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 747324.000000 [07:00:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:05:11] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7210.000000 [07:05:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7226.000000 [07:06:11] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7207.000000 [07:08:11] aliasd on nightshade is CRITICAL: Connection refused [07:08:31] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:08:38] Sun Grid Engine execd on mayapple is UNKNOWN: Execution timeout exceeded [07:10:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:13:04] Maarten Dammers * Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [07:38:19] Sun Grid Engine execd on mayapple is OK: Host and Queues Ok [07:38:19] SSH on mayapple is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [07:39:19] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:39:57] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:40:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:40:11] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1530905.000000 [07:40:11] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:40:11] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1158618.000000 [07:40:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 216824.000000 [07:40:20] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:40:20] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicati [07:40:31] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:10] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21505 MB (3% inode=99%): [07:41:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:41:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:33] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:33] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:33] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:33] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:41:33] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:41:33] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:46:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:46:32] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 566984.000000 [07:46:32] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:11] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [07:47:11] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [07:47:31] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:47:31] / on thyme is UNKNOWN: NRPE: Unable to read output [07:47:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:48:10] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:48:11] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 608 MB (2% inode=93%): [07:48:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 608 MB (2% inode=93%): [07:48:20] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 921959.000000 [07:49:11] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:49:11] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [07:57:10] / on wolfsbane is OK: DISK OK - free space: / 7087 MB (23% inode=93%): [07:57:10] /tmp on wolfsbane is OK: DISK OK - free space: / 7087 MB (23% inode=93%): [07:59:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 314416 MB (5% inode=61%): [08:00:19] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:00:31] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 750924.000000 [08:00:46] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:04:10] Load avg. on wolfsbane is WARNING: WARNING - load average: 4.83, 22.08, 14.41 [08:05:11] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [08:05:11] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10810.000000 [08:05:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10826.000000 [08:06:10] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8979.000000 [08:07:11] Load avg. on wolfsbane is OK: OK - load average: 1.56, 12.84, 12.09 [08:08:11] aliasd on nightshade is CRITICAL: Connection refused [08:10:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:29:31] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3594.000000 [08:31:47] SMTP on sage is OK: SMTP OK - 0.036 sec. response time [08:32:18] NTP on sage is WARNING: NTP WARNING: Server has the LI_ALARM bit set, Offset 0.016732 secs [08:33:31] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1439.000000 [08:35:19] SMTP on z-dat-s5-b is OK: SMTP OK - 0.068 sec. response time [08:39:19] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:40:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:40:11] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1534460.000000 [08:40:11] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:40:11] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1160881.000000 [08:40:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 220424.000000 [08:40:30] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:41:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21333 MB (3% inode=99%): [08:41:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:41:19] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:41:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:41:47] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:41:47] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:42:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:32] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:42:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:42:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:42:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:42:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:31] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:46:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:46:31] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 568332.000000 [08:46:31] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:11] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6160 [08:47:11] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [08:47:19] NTP on sage is OK: NTP OK: Offset 0.00558 secs [08:47:30] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:47:30] / on thyme is UNKNOWN: NRPE: Unable to read output [08:47:37] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:48:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:49:11] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:49:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 925309.000000 [08:49:11] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4816 [08:49:12] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1770.000000 [08:54:11] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3570 [08:58:11] MySQL slave on z-dat-s2-b is OK: Uptime: 3661 Threads: 9 Questions: 5078899 Slow queries: 70 Opens: 26834 Flush tables: 1 Open tables: 256 Queries per second avg: 1387.298 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1650 [09:00:19] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:00:31] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 752404.000000 [09:00:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 314109 MB (5% inode=61%): [09:00:48] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:04:11] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2039.000000 [09:05:10] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [09:06:11] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6955.000000 [09:08:11] aliasd on nightshade is CRITICAL: Connection refused [09:10:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:11:11] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2098 [09:13:11] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1759.000000 [09:31:12] MySQL slave on z-dat-s2-b is OK: Uptime: 5641 Threads: 8 Questions: 7555022 Slow queries: 129 Opens: 36726 Flush tables: 1 Open tables: 256 Queries per second avg: 1339.305 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1729 [09:39:20] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:40:32] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:41:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:41:11] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1537996.000000 [09:41:11] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:41:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 224084.000000 [09:41:11] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1163175.000000 [09:41:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 21135 MB (3% inode=99%): [09:41:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:41:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:41:34] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:41:48] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:41:48] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:42:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:31] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:31] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:31] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:42:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:42:31] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:42:31] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:42:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:42:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:32] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:32] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:32] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:32] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:33] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:33] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:34] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:34] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:35] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:43:36] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:31] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:46:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:46:32] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 568032.000000 [09:46:32] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:47:10] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8250 [09:47:11] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [09:47:32] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:47:32] / on thyme is UNKNOWN: NRPE: Unable to read output [09:48:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:48:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:50:10] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:50:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 928333.000000 [10:00:31] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:00:31] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 752639.000000 [10:00:32] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 313747 MB (5% inode=61%): [10:01:48] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:06:12] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [10:06:12] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6215.000000 [10:08:11] aliasd on nightshade is CRITICAL: Connection refused [10:39:10] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3437 [10:39:10] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3420.000000 [10:40:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:40:30] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:40:31] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:41:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:41:12] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:41:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1541491.000000 [10:41:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1165656.000000 [10:41:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 227684.000000 [10:41:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20951 MB (3% inode=99%): [10:41:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:41:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:41:31] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:41:47] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:41:47] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:42:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:32] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:33] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:42:33] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:32] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:32] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:32] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:32] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:32] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:32] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:32] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:33] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:33] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:34] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:34] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:36] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:31] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:46:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:46:31] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 566504.000000 [10:46:31] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:47:11] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [10:47:30] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:48:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:48:31] / on thyme is UNKNOWN: NRPE: Unable to read output [10:48:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:51:10] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:51:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 931257.000000 [10:51:10] MySQL slave on z-dat-s5-b is OK: Uptime: 10397 Threads: 5 Questions: 85073111 Slow queries: 125 Opens: 1046 Flush tables: 1 Open tables: 255 Queries per second avg: 8182.467 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1769 [10:51:10] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1753.000000 [11:01:30] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:01:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 753045.000000 [11:01:30] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 313279 MB (5% inode=61%): [11:01:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:06:11] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [11:09:11] aliasd on nightshade is CRITICAL: Connection refused [11:17:10] / on wolfsbane is WARNING: DISK WARNING - free space: / 6123 MB (20% inode=93%): [11:17:10] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 6123 MB (20% inode=93%): [11:23:32] / on ortelius is WARNING: DISK WARNING - free space: / 6207 MB (20% inode=92%): [11:40:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:40:31] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:41:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:41:11] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1544981.000000 [11:41:11] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:41:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 231284.000000 [11:41:11] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1168865.000000 [11:41:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:41:19] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:41:30] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:41:47] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:41:48] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:12] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20783 MB (3% inode=99%): [11:42:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:43:30] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:30] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:31] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:43:31] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:31] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:31] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:32] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:33] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:31] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:31] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:31] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:32] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:32] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:33] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:33] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:34] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:34] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:44:35] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:32] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:46:32] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:46:32] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 564626.000000 [11:46:32] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:11] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [11:47:31] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:48:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:48:30] / on thyme is UNKNOWN: NRPE: Unable to read output [11:49:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:51:12] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:51:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 934147.000000 [12:01:31] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:01:31] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 753130.000000 [12:01:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 312935 MB (5% inode=61%): [12:01:47] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:02:56] hi [12:03:06] any lilipond expert here ? [12:06:10] MySQL on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [12:09:11] aliasd on nightshade is CRITICAL: Connection refused [12:09:11] / on wolfsbane is OK: DISK OK - free space: / 13707 MB (45% inode=93%): [12:09:11] /tmp on wolfsbane is OK: DISK OK - free space: / 13707 MB (45% inode=93%): [12:10:11] aliasd on nightshade is OK: TCP OK - 0.010 second response time on port 984 [500 Not found.] [12:23:31] / on ortelius is WARNING: DISK WARNING - free space: / 5773 MB (19% inode=92%): [12:25:30] / on ortelius is OK: DISK OK - free space: / 12533 MB (41% inode=92%): [12:40:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:40:31] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:41:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:41:11] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:41:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 234884.000000 [12:41:11] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1548349.000000 [12:41:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1172013.000000 [12:41:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:41:20] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:41:30] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:41:57] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:41:57] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:42:11] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 20541 MB (3% inode=99%): [12:43:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:43:39] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:30] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:31] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:31] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:31] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:31] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:32] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:32] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:33] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:33] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:34] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:34] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:35] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:38] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:38] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:38] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:38] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:44:38] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:31] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:31] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:31] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:39] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:39] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:46:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70812 MB (11% inode=99%): [12:47:12] MySQL slave on z-dat-s1-b is CRITICAL: Cant connect to MySQL server on z-dat-s1-b (146) [12:47:31] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:47:31] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:47:31] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 562437.000000 [12:47:31] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:47:31] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:10] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:48:31] / on thyme is UNKNOWN: NRPE: Unable to read output [12:49:38] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:52:12] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:52:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 937069.000000 [13:00:11] MySQL on z-dat-s1-b is OK: Uptime: 18090 Threads: 4 Questions: 7875 Slow queries: 2 Opens: 91 Flush tables: 1 Open tables: 80 Queries per second avg: 0.435 [13:01:31] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:01:31] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 753229.000000 [13:01:31] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 312625 MB (5% inode=61%): [13:01:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:41:04] John * [Toolserver-l] enwiki_p corruption [13:41:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1551717.000000 [13:41:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:41:12] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:41:12] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 238484.000000 [13:41:12] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1173562.000000 [13:41:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:41:58] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:41:58] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:42:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:42:19] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:42:28] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:43:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:43:40] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:39] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:40] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:40] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:40] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:41] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:41] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:29] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:29] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:29] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:30] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:30] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:30] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:31] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:32] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:32] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:33] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:33] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:40] floooood [13:46:10] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70604 MB (11% inode=99%): [13:46:29] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:46:29] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:46:29] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:39] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 560095.000000 [13:47:39] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:48:11] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19610 [13:48:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:48:29] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:48:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:48:29] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:40] / on thyme is UNKNOWN: NRPE: Unable to read output [13:49:39] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:52:11] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:52:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 939092.000000 [14:01:39] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 312177 MB (5% inode=61%): [14:01:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:02:29] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:02:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 753868.000000 [14:10:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:40:10] aliasd on nightshade is CRITICAL: Connection refused [14:41:29] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:10] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:42:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 242144.000000 [14:42:11] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1555001.000000 [14:42:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:42:11] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1173285.000000 [14:42:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:42:29] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:42:29] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:42:57] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:42:57] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:44:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:44:38] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:30] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:30] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:37] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:38] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:38] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:38] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:38] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:39] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:39] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:39] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:39] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:40] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:40] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:41] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70430 MB (11% inode=99%): [14:46:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:29] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:29] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:29] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:29] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:29] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:30] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:30] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:31] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:30] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:48:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:48:30] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 557222.000000 [14:48:38] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:48:48] / on thyme is UNKNOWN: NRPE: Unable to read output [14:49:10] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8755 [14:49:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:49:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:53:11] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:53:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 940188.000000 [15:01:49] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 311896 MB (5% inode=61%): [15:01:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:02:29] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:02:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 754312.000000 [15:10:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:18:29] hello all [15:33:10] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3501 [15:40:10] aliasd on nightshade is CRITICAL: Connection refused [15:41:30] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:42:11] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:42:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 245744.000000 [15:42:11] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1172870.000000 [15:42:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:42:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1558320.000000 [15:42:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:42:29] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:42:56] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:42:57] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:43:20] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:44:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:45:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:29] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:29] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:29] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:38] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:38] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:38] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:38] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:38] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:38] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:39] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:39] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:10] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70237 MB (11% inode=99%): [15:46:28] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:30] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:30] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:30] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:31] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:32] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:33] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:33] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:34] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:47:11] MySQL slave on z-dat-s1-b is OK: Uptime: 28111 Threads: 7 Questions: 10816516 Slow queries: 169 Opens: 322 Flush tables: 1 Open tables: 297 Queries per second avg: 384.778 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1768 [15:47:54] Free Memory on turnera is WARNING: WARNING - 6.4% (537192 kB) free! [15:48:49] / on thyme is UNKNOWN: NRPE: Unable to read output [15:49:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:49:29] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:49:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:49:29] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 553644.000000 [15:49:38] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:49:49] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:53:10] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:53:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 942221.000000 [15:54:49] Free Memory on turnera is CRITICAL: CRITICAL - 5.0% (417348 kB) free! [15:55:49] Free Memory on turnera is WARNING: WARNING - 5.3% (448212 kB) free! [16:01:49] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 311398 MB (5% inode=61%): [16:01:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:02:29] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:02:30] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 754478.000000 [16:05:09] DaBPunkt: Morning [16:05:22] that's POV ;) [16:05:48] DaBPunkt: I would assume that it is your POV [16:06:10] no, it is 18:06 here [16:06:26] DaBPunkt: have you had a chance to check your email? [16:06:30] So what can I do for you? [16:07:01] mail? [16:07:49] yeah, enwiki_p is corrupt again [16:09:16] Betacommand: ah on the mailin-list. [16:10:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:10:33] I will send an anser later. In a nutshell: Only 1 of our copy is broken. The other is correct. And the broken one is STILL broken because it was not yet re-setuped. [16:11:57] DaBPunkt: which copy? that way I can avoid querying it [16:13:04] Marc A. Pelletier * Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations [16:30:41] [[Talk:Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=7937&oldid=7368&rcid=21806 * 93.109.15.146 * (+229) () [16:36:36] [[Talk:Main Page]] ! 10https://wiki.toolserver.org/w/index.php?diff=7938&oldid=7937&rcid=21807 * Xaris333 * (-27) (/* CatScan between languages */ ) [16:40:10] aliasd on nightshade is CRITICAL: Connection refused [16:41:29] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:10] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:42:11] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 249344.000000 [16:42:11] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1170795.000000 [16:42:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:42:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:42:28] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:42:57] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:42:57] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:43:11] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1561813.000000 [16:43:29] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:44:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:45:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:29] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:29] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:38] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:38] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:38] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:38] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:38] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:39] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:39] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:40] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:46:38] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:10] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70099 MB (11% inode=99%): [16:47:29] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:30] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:31] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:31] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:32] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:33] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:33] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:34] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:34] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:49] / on thyme is UNKNOWN: NRPE: Unable to read output [16:49:10] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:49:28] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:49:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:49:29] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:50:38] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 550663.000000 [16:50:38] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:51:49] Free Memory on turnera is CRITICAL: CRITICAL - 3.9% (327612 kB) free! [16:53:10] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.549 second response time [16:54:10] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:54:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 944742.000000 [16:55:11] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.022 second response time [17:02:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 311058 MB (5% inode=61%): [17:02:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:03:28] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:03:29] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 755706.000000 [17:09:48] Free Memory on turnera is WARNING: WARNING - 5.3% (441196 kB) free! [17:10:29] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:14:48] Free Memory on turnera is CRITICAL: CRITICAL - 4.8% (403308 kB) free! [17:28:09] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.926 second response time [17:29:08] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.327 second response time [17:39:38] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 104572 MB (10% inode=99%): [17:41:09] aliasd on nightshade is CRITICAL: Connection refused [17:42:28] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:56] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:42:57] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:43:13] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:43:13] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 253004.000000 [17:43:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1169302.000000 [17:43:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:43:19] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:43:28] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:44:09] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1565320.000000 [17:44:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:45:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:45:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:28] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:29] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:29] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:37] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:38] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:38] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:38] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:38] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:38] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:38] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:39] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:39] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:47:28] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69860 MB (11% inode=99%): [17:48:29] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:29] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:29] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:29] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:29] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:29] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:29] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:30] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:31] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:32] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:33] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:48] / on thyme is UNKNOWN: NRPE: Unable to read output [17:50:09] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:50:29] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:50:29] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:50:29] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:50:48] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:51:37] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:51:37] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 547868.000000 [17:55:09] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:55:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 946917.000000 [18:02:48] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 310863 MB (5% inode=61%): [18:02:57] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:03:28] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:03:28] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 757199.000000 [18:05:10] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.856 second response time [18:10:28] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:14:48] Free Memory on turnera is CRITICAL: CRITICAL - 3.9% (322896 kB) free! [18:16:11] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.583 second response time [18:17:09] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.286 second response time [18:41:09] aliasd on nightshade is CRITICAL: Connection refused [18:42:28] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:57] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:42:57] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:43:09] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:43:09] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 256604.000000 [18:43:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1167170.000000 [18:43:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:43:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:43:33] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:44:10] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1568755.000000 [18:44:28] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:45:51] web server (https) throwing 500 Internal Server Error at random [18:45:58] (from nginx) [18:46:09] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:46:29] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:29] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:29] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:46:30] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:38] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:38] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:38] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:38] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:38] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:38] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:39] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:39] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:40] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:47:29] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:28] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:28] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:28] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:28] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:28] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:29] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:29] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:30] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:37] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69684 MB (11% inode=99%): [18:49:28] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:29] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:48] / on thyme is UNKNOWN: NRPE: Unable to read output [18:50:09] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:50:30] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:50:30] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:30] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:50:58] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [18:52:48] eth0 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:52:55] FC 0/17 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Service Check Timed Out) [18:52:56] Sun Grid Engine execd on wolfsbane is CRITICAL: (Service Check Timed Out) [18:52:57] /sql on ptolemy is CRITICAL: (Service Check Timed Out) [19:03:35] DaBPunkt: i cannot log in [19:04:42] [01:54:38 PM] codurrr (mwbot@willow.toolserver.org) left IRC. (Ping timeout: 252 seconds) [19:08:09] i can't log in via putty, known issue? [19:13:17] my ssh connection hasn't dropped, my web requests return 404? [19:16:14] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 546995.000000 [19:16:15] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:16:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69603 MB (11% inode=99%): [19:16:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:16:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:16:16] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:16:16] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:16:16] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:16:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:16:17] / on thyme is UNKNOWN: NRPE: Unable to read output [19:16:32] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:16:32] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:16:32] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:16:32] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:18:05] Free Memory on turnera is OK: OK - 87.6% (7339060 kB) free. [19:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [19:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [19:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [19:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:27:31] what is the right SGE_ROOT= to set in cronie? [19:27:55] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2210.000000 [19:27:56] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2205.000000 [19:28:05] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2211.000000 [19:38:05] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 8.040 second response time [19:43:04] DaB. * Re: [Toolserver-l] enwiki_p corruption [19:48:55] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.592 second response time [19:50:05] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 1.673 second response time [19:51:46] LA2: The same as on the shell: /sge/GE [19:51:55] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3650.000000 [19:51:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3645.000000 [19:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3651.000000 [19:55:54] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3298.000000 [19:55:55] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3580.000000 [19:57:50] I have a cronie on hawthorn and now I set SGE_ROOT=/sge/GE [19:57:57] but I got this back, by e-mail: [19:58:01] error: commlib error: can't connect to service (Connection refused) [19:58:04] Unable to run job: unable to send message to qmaster using port 444 on host "damiana": got send error. [19:58:07] Exiting. [19:58:42] (Earlier I had a different SGE_ROOT and got another error message.) [19:58:55] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1380.000000 [19:59:55] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.524 second response time [20:01:24] [[User:LA2]] ! 10https://wiki.toolserver.org/w/index.php?diff=7939&oldid=6644&rcid=21808 * La2 * (+127) () [20:01:55] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1679.000000 [20:15:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:15:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:15:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:15:26] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 309447 MB (5% inode=61%): [20:15:26] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:15:35] aliasd on nightshade is CRITICAL: Connection refused [20:15:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:35] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1573997.000000 [20:15:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:45] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:15:46] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:46] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:55] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:15:55] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:55] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:56] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 262175.000000 [20:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 952233.000000 [20:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 762868.000000 [20:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:16:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1170233.000000 [20:16:05] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 548458.000000 [20:16:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69300 MB (11% inode=99%): [20:16:15] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:16:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:16:16] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:16] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:17] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:16:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:16:18] / on thyme is UNKNOWN: NRPE: Unable to read output [20:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:25] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:26] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:16:34] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:20:05] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [20:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [20:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [20:22:14] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:28:54] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.701 second response time [20:30:05] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.283 second response time [20:30:54] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.664 second response time [20:38:42] An error occurred while communicating with another application or an upstream server. [20:38:42] There may be more information about this error in the server's error logs. [20:38:42] If you have any queries about this error, please e-mail ts-admins@toolserver.org. [20:40:47] Hi all. Please help: when a cron job runs a python script I get pretty an error after a "import wikipedia" statement. I've uploaded a copy of pywikipedia into my account, and there's a PYTHONPATH=/home/alebot/pywikipedia [20:41:07] statement into my file .environment. [20:42:30] I get a message "ImportError: No module named wikipedia" often, but sometime script runs. [20:43:00] Can any of you guess where such a random error comes from? [20:44:10] Why such an error wasn't raised some time ago? [20:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7251.000000 [20:53:18] alexbrollo: when was the last time you update pywiki? [20:54:49] @Betacommand: yesterday. [20:55:12] are you using custom code? [20:55:48] I'm reviewing scripts just now - I'm reading SGE nightmare. [20:56:40] alexbrollo: a little more detail would be needed to troubleshoot [20:57:46] Ok, consider that I'm a very rough and DIY "programmer" and I feel very unconfortable in UNIX environment. [20:59:04] A (running) job is, to listen to it.wikisource #irc and to write into a text file the output. [21:00:20] A second job is, to read every 30 m this file, to parse it, to select nsPage edits and to "do something" (mainly cleanup). [21:04:31] I'm reading SGE, presently the second script runs as a cronjob "0,30 * * * * cronsub -s cronAlebot $HOME/cronAlebot.py" [21:06:39] I know that script has to be converted into a qcronsub routine; can any of you help? [21:15:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:15:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:15:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:15:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 309433 MB (5% inode=61%): [21:15:25] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [21:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:35] aliasd on nightshade is CRITICAL: Connection refused [21:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:15:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1577450.000000 [21:15:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:45] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:15:46] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:47] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:15:55] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:15:55] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:57] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:57] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 265775.000000 [21:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 952267.000000 [21:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 763233.000000 [21:16:06] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:16:06] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1167759.000000 [21:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 546564.000000 [21:16:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:16:16] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 69109 MB (11% inode=99%): [21:16:16] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:16:16] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:16] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:17] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:16:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:16:18] / on thyme is UNKNOWN: NRPE: Unable to read output [21:16:24] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:24] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:26] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:26] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:26] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:16:34] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:21:34] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [21:21:34] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [21:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [21:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:38:15] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [21:40:15] MySQL slave on cassia is OK: Uptime: 960903 Threads: 9 Questions: 949816645 Slow queries: 13205 Opens: 14187 Flush tables: 1 Open tables: 13973 Queries per second avg: 988.462 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [21:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10851.000000 [22:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:15:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:15:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:15:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:15:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 310038 MB (5% inode=61%): [22:15:25] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:15:35] aliasd on nightshade is CRITICAL: Connection refused [22:15:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:35] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:36] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1580939.000000 [22:15:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:45] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:15:46] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:15:55] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:15:55] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:56] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:56] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 269374.000000 [22:15:57] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 953304.000000 [22:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 764000.000000 [22:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:16:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1165113.000000 [22:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 546269.000000 [22:16:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68924 MB (11% inode=99%): [22:16:15] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:16:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:16:15] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:16] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:16:17] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:16:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:16:18] / on thyme is UNKNOWN: NRPE: Unable to read output [22:16:24] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:26] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:26] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:26] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:16:35] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:16:45] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [22:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [22:21:36] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [22:22:15] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:32:29] @ Betacommand: I see something horrible running :-( .... thanks, I'll ask into toolserver-l. [22:32:32] quit [22:45:15] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [22:47:15] MySQL slave on cassia is OK: Uptime: 964923 Threads: 7 Questions: 953674673 Slow queries: 13316 Opens: 14187 Flush tables: 1 Open tables: 13973 Queries per second avg: 988.342 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [22:52:05] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14451.000000 [23:15:25] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:15:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:15:25] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:15:25] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:15:26] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 309052 MB (5% inode=61%): [23:15:35] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:35] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:35] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:35] aliasd on nightshade is CRITICAL: Connection refused [23:15:35] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:15:35] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:35] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:36] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:45] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:15:45] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1584388.000000 [23:15:45] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:45] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:15:46] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:15:46] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:15:55] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:55] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:55] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:15:55] aliasd on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:55] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:56] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:57] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 272975.000000 [23:16:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 955209.000000 [23:16:05] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 765528.000000 [23:16:05] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:16:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1165033.000000 [23:16:06] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:06] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:15] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 547717.000000 [23:16:15] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:16:15] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 68578 MB (11% inode=99%): [23:16:15] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:16:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:16:16] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:16] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:17] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:16:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc: [23:16:18] / on thyme is UNKNOWN: NRPE: Unable to read output [23:16:25] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:16:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:25] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:25] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:25] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:26] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:26] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:27] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:16:34] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:16:44] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:21:35] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [23:21:35] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [23:21:35] MySQL slave on z-dat-s1-b is CRITICAL: (Return code of 139 is out of bounds) [23:22:14] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:25] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 319883 MB (6% inode=61%): [23:43:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318815 MB (5% inode=61%): [23:45:24] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 321608 MB (6% inode=61%): [23:52:04] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18051.000000