[00:06:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 632510.000000 [00:08:54] aliasd on nightshade is CRITICAL: Connection refused [00:16:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 3909 MB (13% inode=93%): [00:16:52] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3906 MB (13% inode=93%): [00:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:31:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:35:23] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:35:24] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:36:24] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:36:24] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:37:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 748163.000000 [00:38:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:38:24] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7506.000000 [00:38:54] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7273 [00:39:11] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [00:39:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 448247.000000 [00:39:24] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:39:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1170581.000000 [00:39:24] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36813 MB (6% inode=99%): [00:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:40:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:40:54] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85207 [00:40:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:40:54] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:40:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1264775.000000 [00:41:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:41:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:41:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:41:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:41:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:42:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:42:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:42:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:42:23] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:23] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:26] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:42:27] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:43:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:43:31] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:43:31] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:43:31] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:43:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:43:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:25] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:25] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [00:44:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:44:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 346293 MB (6% inode=62%): [00:45:23] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3390.000000 [00:45:53] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2981 [00:48:22] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1624.000000 [00:48:53] MySQL slave on z-dat-s5-b is OK: Uptime: 64000 Threads: 4 Questions: 168526962 Slow queries: 457 Opens: 13337 Flush tables: 1 Open tables: 255 Queries per second avg: 2633.233 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1447 [00:58:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:06:23] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 634837.000000 [01:09:54] aliasd on nightshade is CRITICAL: Connection refused [01:16:55] / on wolfsbane is WARNING: DISK WARNING - free space: / 3814 MB (12% inode=93%): [01:16:56] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3811 MB (12% inode=93%): [01:24:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:31:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:36:23] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:36:23] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:37:24] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:37:25] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:37:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 748440.000000 [01:39:11] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [01:39:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:24] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 449437.000000 [01:39:24] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1169059.000000 [01:39:24] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:39:25] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36745 MB (6% inode=99%): [01:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:40:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:40:54] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 78197 [01:40:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:40:54] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:40:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1268190.000000 [01:41:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:41:40] Susan: do you happen to know if anyone ever did an equivalent of https://en.wikipedia.org/wiki/Wikipedia:Database_reports/User_preferences for dewiki? [01:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:41:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:41:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:42:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:42:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:42:22] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:22] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:42:28] https://en.wikipedia.org/wiki/Wikipedia:Database_reports [01:42:34] I don't see a German equivalent in the sidebar. [01:42:36] So probably not. [01:42:49] If you need a one-time run, I could probably do it. [01:43:24] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:24] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:24] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:24] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:25] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:25] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:26] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:27] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:43:27] / on thyme is UNKNOWN: NRPE: Unable to read output [01:44:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:24] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [01:44:32] Susan: it's not mega important - a german journalist is curious about the prevalence of formal german in the language prefs - but if it doesn't take you too much time... [01:44:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:44:55] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 346195 MB (6% inode=62%): [01:44:55] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2126 [01:45:01] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2140.000000 [01:45:24] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2151.000000 [01:52:08] Susan: oh, actually, james just found it out [01:54:54] 209 users with de-formal on dewiki [01:55:02] Oh, nice. [01:55:16] I had a mysql prompt open, but I got distracted. [01:56:01] I'm running the query now. [01:58:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:58:32] if you have the complete language list, people might still be interested in it in general [01:58:45] could post it on the dewiki village pump [02:03:44] HaeB: Sec. [02:03:58] HaeB: http://p.defau.lt/?f_5ve5U__mUOWkLqc74g1g [02:04:00] There you go. [02:05:54] Free Memory on damiana is WARNING: WARNING - 6.1% (508580 kB) free! [02:06:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 633590.000000 [02:09:54] aliasd on nightshade is CRITICAL: Connection refused [02:12:53] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3629 [02:12:54] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420624 kB) free! [02:13:24] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3658.000000 [02:17:55] / on wolfsbane is WARNING: DISK WARNING - free space: / 3726 MB (12% inode=93%): [02:17:56] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3724 MB (12% inode=93%): [02:23:53] Free Memory on damiana is OK: OK - 8.8% (738536 kB) free. [02:24:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:28:22] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3526.000000 [02:28:52] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3472 [02:31:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:36:23] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:55] MySQL slave on z-dat-s5-b is OK: Uptime: 70480 Threads: 5 Questions: 173465085 Slow queries: 536 Opens: 13937 Flush tables: 1 Open tables: 256 Queries per second avg: 2461.195 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1636 [02:36:55] Free Memory on damiana is WARNING: WARNING - 6.6% (556172 kB) free! [02:37:23] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1410.000000 [02:37:23] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:37:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 748090.000000 [02:38:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:38:21] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:39:05] "formal German"? Distinct enough that it'd have a separate UI language? [02:39:12] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [02:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:40:21] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:40:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1164559.000000 [02:40:21] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 449960.000000 [02:40:21] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:21] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34116 MB (5% inode=99%): [02:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:40:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:41:00] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1647.000000 [02:41:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:41:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:41:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:41:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:41:54] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 68169 [02:41:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:41:54] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:41:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1271687.000000 [02:42:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:42:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:42:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:42:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:42:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:22] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:22] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:22] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:23] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:23] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:23] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:25] / on thyme is UNKNOWN: NRPE: Unable to read output [02:44:31] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:44:31] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:22] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:22] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [02:45:52] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:45:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 346077 MB (6% inode=62%): [02:46:53] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (420316 kB) free! [02:51:20] Susan: https://de.wikipedia.org/wiki/Wikipedia:Fragen_zur_Wikipedia#Statistik_zur_Benutzeroberfl.C3.A4chen-Spracheinstellung [02:51:53] Nice. :-) [02:58:24] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:07:22] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 633449.000000 [03:09:55] aliasd on nightshade is CRITICAL: Connection refused [03:10:52] Free Memory on damiana is WARNING: WARNING - 5.6% (468712 kB) free! [03:18:50] / on wolfsbane is WARNING: DISK WARNING - free space: / 4431 MB (14% inode=93%): [03:18:51] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (418744 kB) free! [03:18:51] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4430 MB (14% inode=93%): [03:21:00] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2120.000000 [03:25:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:29:00] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1701.000000 [03:29:54] Free Memory on damiana is WARNING: WARNING - 5.1% (423200 kB) free! [03:30:51] Free Memory on damiana is CRITICAL: CRITICAL - 4.9% (406796 kB) free! [03:36:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:21] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:38:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:38:23] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:38:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 748450.000000 [03:39:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:40:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [03:40:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:40:22] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 450941.000000 [03:40:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1160870.000000 [03:40:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:40:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36576 MB (5% inode=99%): [03:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:42:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:42:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:42:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:42:51] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:42:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:42:51] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:42:51] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61382 [03:42:51] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:42:52] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:42:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1275112.000000 [03:43:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:43:22] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:43:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:44:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:44:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:21] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:21] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:21] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:21] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:21] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:22] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:22] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:23] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:23] / on thyme is UNKNOWN: NRPE: Unable to read output [03:45:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:45:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:21] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:46:21] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [03:46:50] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:46:51] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345968 MB (6% inode=62%): [03:59:21] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:07:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 634857.000000 [04:09:55] aliasd on nightshade is CRITICAL: Connection refused [04:10:22] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2207.000000 [04:18:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 4250 MB (14% inode=93%): [04:18:52] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4249 MB (14% inode=93%): [04:25:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:30:56] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (397368 kB) free! [04:36:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:38:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 751872.000000 [04:39:23] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:39:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:39:23] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:40:11] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [04:40:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:40:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1163598.000000 [04:40:22] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 453450.000000 [04:40:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:40:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36512 MB (5% inode=99%): [04:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:40:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:42:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:42:21] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [04:42:52] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:42:52] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:42:52] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:42:52] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 54959 [04:42:52] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:42:53] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:42:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1278685.000000 [04:43:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:43:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:43:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:43:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:44:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:44:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:23] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:23] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:24] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:25] / on thyme is UNKNOWN: NRPE: Unable to read output [04:45:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:26] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:27] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:27] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:45:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:46:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [04:46:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:46:55] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345772 MB (6% inode=62%): [04:48:23] s4 replag on z-dat-s5-b is CRITICAL: (Service Check Timed Out) [04:59:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:01:20] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:07:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 638456.000000 [05:10:53] aliasd on nightshade is CRITICAL: Connection refused [05:14:11] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2174.000000 [05:19:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 3977 MB (13% inode=93%): [05:19:52] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 4221 MB (14% inode=93%): [05:25:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:31:53] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (354212 kB) free! [05:36:23] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:38:11] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3614.000000 [05:38:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 755474.000000 [05:39:24] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:39:24] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:39:24] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:41:11] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [05:41:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:41:22] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 455923.000000 [05:41:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1167258.000000 [05:41:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:41:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36394 MB (5% inode=99%): [05:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:42:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:42:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:42:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:42:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:42:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:42:53] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 49888 [05:42:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:42:53] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:42:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1282285.000000 [05:43:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:43:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:43:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:43:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:44:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:44:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:44:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:22] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:24] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:24] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:24] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:24] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:24] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:25] / on thyme is UNKNOWN: NRPE: Unable to read output [05:45:26] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:26] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:31] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:45:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:46:24] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:24] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [05:46:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:47:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345521 MB (6% inode=62%): [05:48:23] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5109.000000 [06:00:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:07:23] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 642056.000000 [06:11:51] aliasd on nightshade is CRITICAL: Connection refused [06:19:51] / on wolfsbane is WARNING: DISK WARNING - free space: / 4038 MB (13% inode=93%): [06:20:53] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3986 MB (13% inode=93%): [06:26:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:31:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:31:53] Free Memory on damiana is CRITICAL: CRITICAL - 4.7% (391832 kB) free! [06:36:54] aliasd on yarrow is CRITICAL: Connection refused [06:37:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:38:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7214.000000 [06:38:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 759075.000000 [06:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:40:24] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:40:24] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:24] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:41:21] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:41:22] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1170858.000000 [06:41:22] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 457693.000000 [06:41:22] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 36215 MB (5% inode=99%): [06:41:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:41:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:41:39] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:42:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:42:11] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [06:42:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:42:53] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:42:53] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:42:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:42:53] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44490 [06:42:53] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:42:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:43:00] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1285892.000000 [06:43:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:44:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:44:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:44:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:24] / on thyme is UNKNOWN: NRPE: Unable to read output [06:45:24] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:24] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:24] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:24] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:25] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:26] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:26] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:27] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:31] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:45:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:47:24] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:24] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [06:47:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:48:23] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8465.000000 [06:48:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 345303 MB (6% inode=62%): [07:00:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:02:22] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 105277 MB (10% inode=99%): [07:08:22] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 645716.000000 [07:12:50] aliasd on nightshade is CRITICAL: Connection refused [07:19:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 3922 MB (13% inode=93%): [07:20:55] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3934 MB (13% inode=93%): [07:27:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:31:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.5% (373388 kB) free! [07:36:55] aliasd on yarrow is CRITICAL: Connection refused [07:37:24] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:38:11] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10815.000000 [07:38:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 762675.000000 [07:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:40:23] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:40:31] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:41:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 460523.000000 [07:41:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1174458.000000 [07:41:23] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35987 MB (5% inode=99%): [07:41:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:41:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:41:23] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:42:11] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [07:42:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:42:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:43:01] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1289492.000000 [07:43:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:43:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:43:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:43:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:43:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:43:55] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35502 [07:43:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:43:56] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:44:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:44:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:44:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:44:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:44:31] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:44:31] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:24] / on thyme is UNKNOWN: NRPE: Unable to read output [07:46:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:24] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:24] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:24] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:24] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:25] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:26] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:26] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:27] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:31] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:46:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:25] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [07:47:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:48:24] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12065.000000 [07:48:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 344897 MB (6% inode=62%): [08:00:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:08:23] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 649316.000000 [08:12:54] aliasd on nightshade is CRITICAL: Connection refused [08:19:54] / on wolfsbane is WARNING: DISK WARNING - free space: / 3602 MB (12% inode=93%): [08:21:52] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3724 MB (12% inode=93%): [08:27:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:32:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.2% (352100 kB) free! [08:37:50] aliasd on yarrow is CRITICAL: Connection refused [08:38:21] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:39:10] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14474.000000 [08:39:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:39:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 766335.000000 [08:40:23] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:40:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:40:31] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:41:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 464123.000000 [08:41:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1178058.000000 [08:41:23] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35810 MB (5% inode=99%): [08:41:24] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:24] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:24] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:42:11] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [08:42:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:42:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:43:10] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:43:25] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:44:01] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1293151.000000 [08:44:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:44:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:44:31] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:44:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:44:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:44:55] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:44:55] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 25519 [08:44:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:44:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:46:23] / on thyme is UNKNOWN: NRPE: Unable to read output [08:46:24] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:24] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:25] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:25] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:25] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:25] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:26] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:26] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:46:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:47:24] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:24] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:24] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [08:47:54] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:48:25] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15665.000000 [08:49:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 344040 MB (6% inode=62%): [09:01:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:01:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:08:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 652917.000000 [09:13:51] aliasd on nightshade is CRITICAL: Connection refused [09:19:53] / on wolfsbane is WARNING: DISK WARNING - free space: / 3458 MB (11% inode=93%): [09:22:53] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3560 MB (11% inode=93%): [09:27:24] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2155 [09:27:54] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2193.000000 [09:27:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:32:53] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (363416 kB) free! [09:37:52] aliasd on yarrow is CRITICAL: Connection refused [09:38:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:39:11] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18074.000000 [09:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 769935.000000 [09:40:22] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:40:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:41:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 467723.000000 [09:41:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1181658.000000 [09:41:23] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35613 MB (5% inode=99%): [09:41:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:23] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:42:11] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [09:42:12] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:42:31] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:44:00] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1296752.000000 [09:44:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:44:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:44:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:44:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:45:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:45:31] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:45:31] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:45:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:45:54] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17030 [09:45:54] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:45:54] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:46:30] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:46:30] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:21] / on thyme is UNKNOWN: NRPE: Unable to read output [09:47:22] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:22] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:22] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:22] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:22] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:22] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:23] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:24] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:47:24] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:48:23] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:23] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:23] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [09:48:52] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:49:25] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19325.000000 [09:50:51] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 343628 MB (6% inode=62%): [09:51:53] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3606.000000 [09:52:22] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3628 [10:01:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:02:20] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:09:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 656576.000000 [10:13:55] aliasd on nightshade is CRITICAL: Connection refused [10:18:55] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3293 MB (10% inode=93%): [10:18:55] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3293 MB (10% inode=93%): [10:19:53] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3392 MB (11% inode=93%): [10:20:53] / on wolfsbane is WARNING: DISK WARNING - free space: / 3368 MB (11% inode=93%): [10:28:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:30:22] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3510 [10:30:53] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 16052 MB (1% inode=99%): [10:30:54] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3376.000000 [10:32:53] Free Memory on damiana is CRITICAL: CRITICAL - 4.3% (361832 kB) free! [10:37:55] aliasd on yarrow is CRITICAL: Connection refused [10:38:25] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:39:39] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:39:54] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3277 MB (10% inode=93%): [10:39:54] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3277 MB (10% inode=93%): [10:39:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 773535.000000 [10:40:11] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21734.000000 [10:40:23] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:40:52] / on wolfsbane is WARNING: DISK WARNING - free space: / 3321 MB (11% inode=93%): [10:40:53] /tmp on wolfsbane is WARNING: DISK WARNING - free space: / 3321 MB (11% inode=93%): [10:41:27] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 471323.000000 [10:41:27] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1185258.000000 [10:41:27] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:27] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:27] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35385 MB (5% inode=99%): [10:41:27] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:31] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:41:40] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:42:22] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:42:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:43:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [10:43:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:44:11] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:44:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:44:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:44:55] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:45:11] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1300413.000000 [10:45:30] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:45:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:45:30] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:45:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:45:54] MySQL slave on z-dat-s1-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8095 [10:45:55] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3279 MB (10% inode=93%): [10:45:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:45:55] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3279 MB (10% inode=93%): [10:45:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:46:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:46:25] MySQL slave on daphne is OK: Uptime: 658783 Threads: 68 Questions: 87701690 Slow queries: 138906 Opens: 28571 Flush tables: 1 Open tables: 811 Queries per second avg: 133.126 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1756 [10:46:31] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:32] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:32] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:46:51] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1680.000000 [10:47:23] / on thyme is UNKNOWN: NRPE: Unable to read output [10:47:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:23] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:23] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:24] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:25] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:25] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:47:26] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:23] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:24] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:25] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:25] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [10:48:55] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:50:23] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22985.000000 [10:50:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 343367 MB (6% inode=62%): [10:58:53] MySQL slave on z-dat-s1-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3204 [11:02:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:04:54] MySQL slave on z-dat-s1-b is OK: Uptime: 101658 Threads: 11 Questions: 103584090 Slow queries: 1396 Opens: 662 Flush tables: 1 Open tables: 379 Queries per second avg: 1018.946 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1780 [11:10:23] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 660236.000000 [11:13:54] aliasd on nightshade is CRITICAL: Connection refused [11:29:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:31:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:32:52] Free Memory on damiana is CRITICAL: CRITICAL - 4.1% (347560 kB) free! [11:37:55] aliasd on yarrow is CRITICAL: Connection refused [11:39:22] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:39:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 777135.000000 [11:40:11] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25334.000000 [11:40:23] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:40:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:22] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 474923.000000 [11:41:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1188858.000000 [11:41:23] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 35129 MB (5% inode=99%): [11:41:23] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:41:39] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:42:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:42:30] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:43:10] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:43:10] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [11:44:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:44:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:44:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:45:24] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [11:45:31] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:45:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:45:31] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:01] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1304073.000000 [11:46:30] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:46:54] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:46:55] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3078 MB (10% inode=93%): [11:46:55] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:46:55] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:55] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 3077 MB (10% inode=93%): [11:47:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:23] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:47:23] / on thyme is UNKNOWN: NRPE: Unable to read output [11:47:23] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:23] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:23] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:47:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:25] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:25] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:25] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:26] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:26] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:48:27] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [11:49:53] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:51:23] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26645.000000 [11:51:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 342816 MB (6% inode=62%): [12:02:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:10:24] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 663836.000000 [12:14:55] aliasd on nightshade is CRITICAL: Connection refused [12:29:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:31:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:32:53] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (331136 kB) free! [12:38:53] aliasd on yarrow is CRITICAL: Connection refused [12:39:40] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:39:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 780735.000000 [12:40:11] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28935.000000 [12:40:22] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:40:23] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:40:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:23] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 478523.000000 [12:41:23] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1192458.000000 [12:41:23] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34866 MB (5% inode=99%): [12:41:23] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:30] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:41:46] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:42:31] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:43:11] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [12:43:11] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:43:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:12] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:45:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:45:30] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:31] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:31] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:45:54] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:45:54] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:46:01] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1307674.000000 [12:47:23] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:24] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:24] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:24] / on thyme is UNKNOWN: NRPE: Unable to read output [12:47:24] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:24] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:24] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:25] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:31] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:31] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:31] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:47:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:47:53] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2804 MB (9% inode=93%): [12:47:53] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:47:53] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:47:53] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2804 MB (9% inode=93%): [12:48:24] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:24] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:24] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:24] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:24] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:24] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:48:25] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:23] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:49:23] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [12:50:51] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:52:22] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30305.000000 [12:52:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 342207 MB (6% inode=62%): [13:02:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:11:21] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 667496.000000 [13:15:51] aliasd on nightshade is CRITICAL: Connection refused [13:27:21] @replag [13:27:23] Betacommand: s1-rr-a-wd: 1w 7h 17m 32s [+1.00 s/s]; s1-user-wd: 1w 6d 20h 28s [+1.00 s/s]; s2-rr: 22m 4s [+0.10 s/s]; s2-user: 22m 4s [+0.10 s/s]; s2-user-c: error; s2-user-wd: 1w 2d 1h 39m 50s [+1.00 s/s]; s3-user: 16s [-0.00 s/s]; s3-user-wd: 1w 22h 26m 13s [+1.00 s/s] [13:27:23] Betacommand: s4-user-wd: 2w 1d 3h 56m 0s [+1.00 s/s]; s5-rr-a-wd: 5d 13h 41m 37s [+1.00 s/s]; s5-user: 3m 52s [-0.06 s/s]; s5-user-c: 9h 15s [+1.00 s/s]; s6-user-wd: 1w 1d 3h 27m 34s [+1.00 s/s]; s7-user-wd: 1w 17h 41m 11s [+1.00 s/s] [13:28:39] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:28:45] FC 0/23 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:28:45] FC 0/21 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:28:45] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:28:45] FC 0/18 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:28:46] FC 0/22 on fsw1-n1-oe16-esams.mgmt is CRITICAL: (Return code of 142 is out of bounds) [13:31:11] SSH to both nightshade and willow seems to be down [13:31:41] I take that back, just really really really slow [13:34:58] /sql on z-dat-s1-b is CRITICAL: (Service Check Timed Out) [13:34:58] Free Memory on damiana is CRITICAL: (Service Check Timed Out) [13:34:58] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 106181 MB (10% inode=99%): [13:34:58] Sun Grid Engine execd on wolfsbane is CRITICAL: (Service Check Timed Out) [13:34:58] Sun Grid Engine execd on ortelius is CRITICAL: (Service Check Timed Out) [13:34:58] Free Memory on damiana is WARNING: WARNING - 5.7% (476340 kB) free! [13:35:10] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:35:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:37:11] SMTP on ortelius is OK: SMTP OK - 0.003 sec. response time [13:37:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:37:31] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.005 second response time [13:39:21] aliasd on yarrow is CRITICAL: Connection refused [13:39:51] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:40:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:40:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32559.000000 [13:40:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 784379.000000 [13:40:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:41:40] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1196081.000000 [13:41:50] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 482151.000000 [13:41:51] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:41:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34639 MB (5% inode=99%): [13:41:52] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:20] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [13:43:20] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:43:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:32] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:40] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:43:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:43:52] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:22] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:22] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:44:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:44:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:45:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:46:01] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:46:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:46:01] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:46:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:46:20] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1311291.000000 [13:47:33] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:47:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:33] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:51] / on thyme is UNKNOWN: NRPE: Unable to read output [13:48:01] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:01] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2659 MB (8% inode=93%): [13:48:02] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:02] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:48:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:48:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2664 MB (8% inode=93%): [13:48:20] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:20] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:48:20] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:48:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:55] Coren: You around? [13:49:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:41] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:49:41] Ayup. [13:50:05] I was wondering if you had some time today to help me get my python app running [13:50:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [13:50:25] greenrosetta: I will, this afternoon (ETD) [13:50:28] EDT* [13:50:36] ok [13:51:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:52:30] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33917.000000 [13:53:02] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 341900 MB (6% inode=62%): [14:02:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:11:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 671115.000000 [14:16:11] aliasd on nightshade is CRITICAL: Connection refused [14:37:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:39:22] aliasd on yarrow is CRITICAL: Connection refused [14:39:51] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:40:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36159.000000 [14:40:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:40:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 787979.000000 [14:41:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1199681.000000 [14:41:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 485752.000000 [14:41:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34363 MB (5% inode=99%): [14:41:52] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:41:52] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:21] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [14:43:21] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:43:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:43:31] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:43:31] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:43:32] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:43:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:44:21] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:44:41] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:44:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:44:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:21] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:45:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:45:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:46:02] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:46:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:46:02] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:46:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1314891.000000 [14:47:32] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:47:32] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:51] / on thyme is UNKNOWN: NRPE: Unable to read output [14:48:01] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:01] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2451 MB (8% inode=93%): [14:48:01] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:48:21] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:48:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:02] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:49:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:49:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2439 MB (8% inode=93%): [14:49:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:12] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:49:41] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:51:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [14:52:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:52:32] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37517.000000 [14:53:02] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 341526 MB (6% inode=62%): [15:03:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:11:10] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2119 [15:11:41] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 674715.000000 [15:16:11] aliasd on nightshade is CRITICAL: Connection refused [15:16:27] @replag [15:16:30] Betacommand: s1-rr-a-wd: 1w 9h 6m 38s [+1.00 s/s]; s1-user-wd: 1w 6d 21h 49m 34s [+1.00 s/s]; s2-rr: 38m 6s [+0.15 s/s]; s2-user: 38m 6s [+0.15 s/s]; s2-user-c: error; s2-user-wd: 1w 2d 3h 28m 56s [+1.00 s/s]; s3-user: 10s [-0.00 s/s]; s3-user-wd: 1w 1d 15m 19s [+1.00 s/s] [15:16:31] Betacommand: s4-user-wd: 2w 1d 5h 45m 7s [+1.00 s/s]; s5-rr-a-wd: 5d 15h 30m 44s [+1.00 s/s]; s5-user-c: 10h 49m 22s [+1.00 s/s]; s6-user-wd: 1w 1d 5h 16m 41s [+1.00 s/s]; s7-user-wd: 1w 19h 30m 18s [+1.00 s/s] [15:37:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:39:51] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:40:20] aliasd on yarrow is CRITICAL: Connection refused [15:40:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:40:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39760.000000 [15:40:42] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 791579.000000 [15:40:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:41:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1203280.000000 [15:41:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 489352.000000 [15:41:52] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 34097 MB (5% inode=99%): [15:41:52] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:42:51] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:43:32] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:43:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:43:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:43:52] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:44:22] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [15:44:22] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:44:22] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:44:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:44:41] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:44:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:20] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:45:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:45:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:46:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:47:01] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:47:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:47:02] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:47:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:47:20] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1318551.000000 [15:47:32] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:47:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:32] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:47:51] / on thyme is UNKNOWN: NRPE: Unable to read output [15:48:02] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:02] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2206 MB (7% inode=93%): [15:48:02] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:48:21] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:48:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:49:01] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:10] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:49:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:49:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 2183 MB (7% inode=93%): [15:49:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:11] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3659 [15:49:12] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:49:12] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:50:40] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:51:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [15:52:32] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41117.000000 [15:53:00] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:54:01] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 341135 MB (6% inode=62%): [15:58:48] enwiki_p is trashed\ [16:03:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:11:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 678316.000000 [16:16:10] aliasd on nightshade is CRITICAL: Connection refused [16:25:59] @replag [16:26:00] jeremyb_: s1-rr-a-wd: 1w 10h 16m 8s [+1.00 s/s]; s1-user: 18s [-0.00 s/s]; s1-user-wd: 1w 6d 22h 59m 4s [+1.00 s/s]; s2-rr: 1h 23m 35s [+0.65 s/s]; s2-user: 1h 23m 35s [+0.65 s/s]; s2-user-c: error; s2-user-wd: 1w 2d 4h 38m 26s [+1.00 s/s]; s3-user-wd: 1w 1d 1h 24m 49s [+1.00 s/s] [16:26:01] jeremyb_: s4-user-wd: 2w 1d 6h 54m 38s [+1.00 s/s]; s5-rr-a-wd: 5d 16h 40m 15s [+1.00 s/s]; s5-user-c: 11h 58m 53s [+1.00 s/s]; s6-user-wd: 1w 1d 6h 26m 12s [+1.00 s/s]; s7-user-wd: 1w 20h 39m 49s [+1.00 s/s] [16:34:11] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3394 [16:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:37:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:40:22] aliasd on yarrow is CRITICAL: Connection refused [16:40:32] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43360.000000 [16:40:42] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:40:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 795189.000000 [16:40:52] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:41:11] MySQL slave on z-dat-s2-b is OK: Uptime: 207958 Threads: 10 Questions: 195078846 Slow queries: 2912 Opens: 2021935 Flush tables: 1 Open tables: 256 Queries per second avg: 938.68 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1701 [16:41:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1206881.000000 [16:41:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 492952.000000 [16:41:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33823 MB (5% inode=99%): [16:41:52] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:42:51] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:21] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [16:44:21] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:31] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:31] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:31] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:41] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:44:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:45:21] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:45:21] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:46:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:46:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:46:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:46:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:47:01] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:47:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:47:02] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1322150.000000 [16:47:32] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:47:32] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:47:51] / on thyme is UNKNOWN: NRPE: Unable to read output [16:48:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:48:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:48:21] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:48:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:01] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:01] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 1658 MB (5% inode=93%): [16:49:01] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:49:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:49:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 1659 MB (5% inode=93%): [16:49:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:49:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:01] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:50:41] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:51:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [16:52:32] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44717.000000 [16:53:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:54:01] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340476 MB (6% inode=62%): [17:03:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:11:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 681917.000000 [17:16:11] aliasd on nightshade is CRITICAL: Connection refused [17:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:38:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:40:20] aliasd on yarrow is CRITICAL: Connection refused [17:41:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:41:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47020.000000 [17:41:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:41:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1210481.000000 [17:41:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 496552.000000 [17:41:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33591 MB (5% inode=99%): [17:41:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 798849.000000 [17:41:52] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:41:53] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:42:52] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:21] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [17:44:22] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:31] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:31] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:31] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:44:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:42] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:44:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:45:20] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:45:20] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:46:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:46:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:47:01] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:47:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:47:02] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:47:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:47:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1325750.000000 [17:47:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:47:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:47:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:32] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:51] / on thyme is UNKNOWN: NRPE: Unable to read output [17:48:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:48:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:22] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:48:32] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:49:02] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:02] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 1447 MB (4% inode=93%): [17:49:02] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:49:12] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 1448 MB (4% inode=93%): [17:49:12] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:49:12] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:12] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:49:12] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:50:01] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:50:32] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:50:42] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:51:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [17:52:31] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48317.000000 [17:54:01] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 340191 MB (6% inode=62%): [17:54:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:04:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:11:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 685516.000000 [18:16:11] aliasd on nightshade is CRITICAL: Connection refused [18:17:11] aliasd on nightshade is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [18:17:21] aliasd on yarrow is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [18:36:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:38:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:42:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50680.000000 [18:42:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:42:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1214141.000000 [18:42:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 500211.000000 [18:42:55] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33342 MB (5% inode=99%): [18:42:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 802509.000000 [18:42:55] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:42:55] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:42:55] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:44:22] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [18:44:22] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:32] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:44:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:41] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:44:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:45:20] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:45:20] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:46:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:46:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:47:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1329351.000000 [18:47:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:47:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:47:51] / on thyme is UNKNOWN: NRPE: Unable to read output [18:48:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:48:02] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:48:02] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:48:25] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:25] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:48:31] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:31] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:49:01] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:02] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 1251 MB (4% inode=93%): [18:49:02] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:10] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:49:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 1256 MB (4% inode=93%): [18:49:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:49:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:49:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:50:01] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:10] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:10] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:50:41] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:51:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [18:52:31] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51917.000000 [18:55:01] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 339758 MB (6% inode=62%): [18:55:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:02:40] /sql on z-dat-s1-b is WARNING: DISK WARNING - free space: /sql 106272 MB (10% inode=99%): [19:04:17] Coren: [19:04:31] I am. I have a meeting, but then I'm all yours. [19:04:39] sweet [19:04:46] hello [19:04:52] just say my name so I get the beep please [19:04:52] maintenance tonight [19:04:57] greenrosetta: [19:05:06] not u nosy [19:05:07] :D [19:05:11] ok :D [19:05:36] going to prepare [19:05:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:06:47] hmmm. why is the server refuisng my key? [19:12:41] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 689176.000000 [19:13:07] is there aproblem with connectiong to tools directly? [19:20:56] ssh into tool-server now gives me "Permission denied (publickey). [19:20:56] " [19:23:06] greenrosetta: I'm able to log into willow. [19:23:36] what is the full bname for willow? I am logged into bastion right now [19:23:47] willow.toolserver.org [19:26:00] ok going to take s1-user offline [19:32:31] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [19:32:35] johang: can you ssh to bastion, and from there ssh to willow? [19:32:51] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [19:32:51] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [19:33:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [19:35:07] greenrosetta: what's bastion's full hostname? [19:35:35] bastion.wmflabs.org [19:36:13] I can't log in to wmflabs. don't have an account there. [19:36:23] only ts [19:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:38:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:42:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:42:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54279.000000 [19:42:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:42:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [19:43:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 33086 MB (5% inode=99%): [19:43:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 503872.000000 [19:43:51] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:43:51] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:51] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:43:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 806169.000000 [19:44:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:44:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:41] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:44:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:20] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [19:45:21] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:21] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:21] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:45:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:31] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:45:32] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:46:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:47:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1332951.000000 [19:47:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:47:31] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:48:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:22] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:48:31] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:31] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:51] / on thyme is UNKNOWN: NRPE: Unable to read output [19:49:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:49:02] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:49:03] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:03] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:03] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 1012 MB (3% inode=93%): [19:49:03] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:03] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:49:32] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:50:02] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:50:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 1004 MB (3% inode=93%): [19:50:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:50:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:50:41] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:51:22] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [19:52:31] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55517.000000 [19:55:01] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 339338 MB (6% inode=62%): [19:55:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:56:10] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 62215 MB (6% inode=99%): [20:03:11] /sql on rosemary is OK: DISK OK - free space: /sql 108081 MB (11% inode=99%): [20:06:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:13:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 692836.000000 [20:28:21] /sql/data/commonswiki on rosemary is CRITICAL: DISK CRITICAL - /sql/data/commonswiki is not accessible: No such file or directory [20:30:20] /sql/data/commonswiki on rosemary is OK: DISK OK - free space: /sql/data/commonswiki 136721 MB (22% inode=99%): [20:32:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [20:32:50] MySQL on rosemary is OK: Uptime: 13 Threads: 20 Questions: 67 Slow queries: 0 Opens: 19 Flush tables: 1 Open tables: 12 Queries per second avg: 5.153 [20:33:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3989.000000 [20:33:48] ok s1-user is back [20:33:50] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3998 [20:33:53] maintenance done :) [20:39:02] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:39:51] Coren: back? [20:42:30] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56032.000000 [20:43:37] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3595.000000 [20:43:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:43:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1219824.000000 [20:43:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 507189.000000 [20:43:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32837 MB (5% inode=99%): [20:43:52] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 809344.000000 [20:43:52] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:43:52] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:52] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:43:52] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3526 [20:44:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:41] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:44:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:45:57] greenrosetta: Back now. [20:46:05] excellent [20:46:19] so my first problem is I can't acess toolserver now [20:46:21] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [20:46:21] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:21] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:21] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:46:30] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:30] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:30] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:30] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:34] I was able to SSH in dorectrly a few days ago [20:46:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:46:46] i can SSH into bastion just fine [20:46:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:46:53] shouldnt I be able to ssh tools-login from there? [20:47:23] greenrosetta: if you make certain that you forward your agent, yes. [20:47:38] no idea on how to do that [20:47:40] greenrosetta: But, in general, it's simpler to just ssh directly to tools-login.wmflabs.org :-) [20:47:46] yes, thats what I did the other day [20:47:49] Also, we probably want to talk on #wikimedia-labs [20:47:53] ah.. [20:47:56] my bad [20:48:00] and please don't call the wmf labs servers 'the toolserver' :-) [20:48:05] it's confusing [20:48:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1336495.000000 [20:48:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:48:21] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:48:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:48:31] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:48:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:48:31] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:48:50] / on thyme is UNKNOWN: NRPE: Unable to read output [20:49:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:49:06] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:49:06] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:06] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:06] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 725 MB (2% inode=93%): [20:49:06] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:06] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:49:32] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:01] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:50:41] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:10] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:51:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 745 MB (2% inode=93%): [20:51:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:51:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:51:21] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [20:52:31] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54074.000000 [20:55:01] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 339035 MB (6% inode=62%): [20:55:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:58:32] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1795.000000 [20:58:51] MySQL slave on rosemary is OK: Uptime: 1580 Threads: 11 Questions: 1550328 Slow queries: 226 Opens: 457 Flush tables: 1 Open tables: 326 Queries per second avg: 981.220 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1742 [21:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:06:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:13:41] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 693467.000000 [21:32:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7570.000000 [21:39:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:42:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:42:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40679.000000 [21:43:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [21:43:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1211351.000000 [21:43:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 508777.000000 [21:43:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32549 MB (5% inode=99%): [21:43:51] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 811368.000000 [21:43:51] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:43:51] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:43:52] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:44:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:44:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:42] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:44:52] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:22] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:46:22] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:22] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:32] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [21:46:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:32] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:47:41] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1339751.000000 [21:48:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [21:48:22] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:22] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:48:32] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:32] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:48:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:32] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:52] / on thyme is UNKNOWN: NRPE: Unable to read output [21:49:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [21:49:02] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [21:49:02] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:02] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:02] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 546 MB (1% inode=93%): [21:49:03] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:03] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:49:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:01] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:50:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [21:51:12] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 539 MB (1% inode=93%): [21:51:12] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:51:12] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:12] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:12] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:20] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:51:42] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:53:32] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36784.000000 [21:55:01] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 336706 MB (6% inode=62%): [21:55:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:07:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:13:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 692199.000000 [22:33:32] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11229.000000 [22:37:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:39:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:42:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:42:31] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17109.000000 [22:43:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1203163.000000 [22:43:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:43:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32359 MB (5% inode=99%): [22:43:51] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 508899.000000 [22:43:51] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 811149.000000 [22:43:51] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:43:51] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:43:51] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:44:40] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:41] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:41] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:41] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:44:52] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:20] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:46:21] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:21] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:30] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:46:31] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:31] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:31] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:30] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:50] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:48:40] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:20] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1343088.000000 [22:49:20] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:49:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:21] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:30] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:49:30] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:30] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:49:30] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:31] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:31] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:51] / on thyme is UNKNOWN: NRPE: Unable to read output [22:50:01] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:50:01] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:50:01] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:01] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:01] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 379 MB (1% inode=93%): [22:50:01] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:02] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:50:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:30] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:51:41] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:52:11] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:52:11] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 360 MB (1% inode=93%): [22:52:11] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:52:11] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:52:11] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:52:11] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:52:20] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:54:30] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13220.000000 [22:55:00] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 336368 MB (6% inode=62%): [22:55:01] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:07:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:14:40] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 691226.000000 [23:17:30] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3097.000000 [23:21:30] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1171.000000 [23:28:31] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3218.000000 [23:32:31] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 840.000000 [23:34:30] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14890.000000 [23:39:01] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:42:31] SMTP on z-dat-s5-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1195053.000000 [23:43:41] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:43:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 32206 MB (5% inode=99%): [23:43:51] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 810556.000000 [23:43:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 508849.000000 [23:43:52] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:43:52] SMTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:52] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:44:41] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:42] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:44:51] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:31] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:46:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:32] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:32] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:21] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:47:21] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:21] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:32] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:48:42] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1346274.000000 [23:49:21] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:49:21] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:21] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:31] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:49:31] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:31] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:49:31] SMTP on z-dat-s1-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:31] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:32] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:49:51] / on thyme is UNKNOWN: NRPE: Unable to read output [23:50:00] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:50:02] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:50:02] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:02] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:02] / on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 205 MB (0% inode=93%): [23:50:02] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:02] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:50:02] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:31] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:51:41] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:52:12] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:52:12] /tmp on wolfsbane is CRITICAL: DISK CRITICAL - free space: / 195 MB (0% inode=93%): [23:52:12] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:52:12] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:52:12] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:52:12] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:52:22] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:54:50] CAM on hemlock is OK: OK - cam detected no new errors [23:55:01] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 334858 MB (6% inode=61%): [23:55:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output