[00:00:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:11:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:18:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:18:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 440011 MB (8% inode=64%): [00:21:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:21:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:56] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 93047 [00:22:56] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 107921.000000 [00:24:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77897.000000 [00:24:05] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71083.000000 [00:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59403.000000 [00:28:13] aliasd on nightshade is CRITICAL: Connection refused [00:30:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 195929.000000 [00:31:15] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:31:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [00:32:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [00:33:43] / on thyme is UNKNOWN: NRPE: Unable to read output [00:33:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [00:34:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 92885.000000 [00:34:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:34:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:35:34] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [00:35:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [00:36:05] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [00:36:05] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [00:36:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [00:37:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [00:37:33] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [00:40:04] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [00:41:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40551.000000 [00:43:25] aliasd on yarrow is CRITICAL: Connection refused [00:50:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82830 MB (13% inode=99%): [00:55:05] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:00:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:10:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 31828 MB (3% inode=99%): [01:11:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:17:55] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2194.000000 [01:18:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:56] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:21:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 435049 MB (8% inode=64%): [01:21:25] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:23:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 91651 [01:23:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110738.000000 [01:25:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81557.000000 [01:25:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74743.000000 [01:26:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62487.000000 [01:28:14] aliasd on nightshade is CRITICAL: Connection refused [01:31:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 198044.000000 [01:31:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [01:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:32:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [01:33:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [01:34:43] / on thyme is UNKNOWN: NRPE: Unable to read output [01:34:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [01:35:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91369.000000 [01:35:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:35:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:36:33] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [01:36:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [01:37:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [01:37:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [01:37:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [01:37:54] / on willow is WARNING: DISK WARNING - free space: / 16356 MB (15% inode=98%): [01:38:05] /tmp on willow is WARNING: DISK WARNING - free space: / 16223 MB (15% inode=98%): [01:38:17] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [01:38:23] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [01:41:03] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [01:41:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3634.000000 [01:42:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40556.000000 [01:43:03] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 11457 MB (10% inode=98%): [01:43:25] aliasd on yarrow is CRITICAL: Connection refused [01:43:53] / on willow is CRITICAL: DISK CRITICAL - free space: / 10490 MB (10% inode=97%): [01:51:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82537 MB (13% inode=99%): [01:56:05] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [02:00:05] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:03:53] / on willow is OK: DISK OK - free space: / 30370 MB (29% inode=99%): [02:04:04] /tmp on willow is OK: DISK OK - free space: / 30369 MB (29% inode=99%): [02:11:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:19:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:19:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:25] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:22:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:22:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 470008 MB (8% inode=64%): [02:23:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 89178 [02:23:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112136.000000 [02:25:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85157.000000 [02:25:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78343.000000 [02:26:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65916.000000 [02:28:14] aliasd on nightshade is CRITICAL: Connection refused [02:31:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 198829.000000 [02:31:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [02:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:32:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [02:33:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [02:34:44] / on thyme is UNKNOWN: NRPE: Unable to read output [02:34:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [02:35:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88855.000000 [02:35:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:35:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:36:33] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [02:36:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [02:37:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [02:37:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [02:37:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [02:38:18] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [02:38:25] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [02:41:05] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [02:41:54] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7234.000000 [02:42:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40638.000000 [02:44:24] aliasd on yarrow is CRITICAL: Connection refused [02:51:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82398 MB (13% inode=99%): [02:56:04] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:00:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:12:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:19:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:19:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:21:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473812 MB (8% inode=64%): [03:24:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 88285 [03:24:54] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114735.000000 [03:25:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88757.000000 [03:25:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81942.000000 [03:27:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69172.000000 [03:28:14] aliasd on nightshade is CRITICAL: Connection refused [03:31:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 201432.000000 [03:31:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [03:32:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [03:34:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [03:35:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88105.000000 [03:35:04] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:35:04] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:35:43] / on thyme is UNKNOWN: NRPE: Unable to read output [03:35:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [03:36:33] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [03:36:52] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [03:37:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [03:37:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [03:38:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [03:38:26] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [03:39:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [03:41:04] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [03:41:54] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10834.000000 [03:43:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41720.000000 [03:44:24] aliasd on yarrow is CRITICAL: Connection refused [03:52:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82196 MB (13% inode=99%): [03:57:04] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:57:55] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2198.000000 [04:00:04] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:12:42] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:19:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:19:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:21:25] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:21:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3638.000000 [04:22:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 472968 MB (8% inode=64%): [04:24:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 85498 [04:24:55] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118335.000000 [04:25:06] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 92358.000000 [04:25:06] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85545.000000 [04:27:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72287.000000 [04:29:13] aliasd on nightshade is CRITICAL: Connection refused [04:31:34] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [04:32:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 204577.000000 [04:32:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [04:35:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86089.000000 [04:35:05] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:35:05] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:35:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [04:35:43] / on thyme is UNKNOWN: NRPE: Unable to read output [04:35:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [04:36:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [04:37:06] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [04:37:06] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [04:37:34] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [04:38:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [04:38:35] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [04:39:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [04:41:13] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [04:42:54] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14494.000000 [04:44:24] aliasd on yarrow is CRITICAL: Connection refused [04:44:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45150.000000 [04:53:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82025 MB (13% inode=99%): [04:57:04] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:00:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:12:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:19:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:19:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:21:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7238.000000 [05:22:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:22:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473911 MB (8% inode=64%): [05:22:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:25:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 86570 [05:25:53] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121995.000000 [05:26:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 96018.000000 [05:26:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 89205.000000 [05:27:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75891.000000 [05:29:14] aliasd on nightshade is CRITICAL: Connection refused [05:31:34] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [05:32:15] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [05:33:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 208236.000000 [05:35:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:35:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:35:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [05:35:44] / on thyme is UNKNOWN: NRPE: Unable to read output [05:35:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [05:36:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86807.000000 [05:36:56] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [05:37:43] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [05:38:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [05:38:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [05:38:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [05:38:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [05:39:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [05:41:14] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [05:42:54] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18094.000000 [05:44:24] aliasd on yarrow is CRITICAL: Connection refused [05:44:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48751.000000 [05:53:15] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:53:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81828 MB (13% inode=99%): [05:57:14] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:00:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:12:54] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 40412 MB (7% inode=99%): [06:13:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:20:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:20:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:21:33] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:22:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10899.000000 [06:23:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:23:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473773 MB (8% inode=64%): [06:23:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 82530 [06:26:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99618.000000 [06:26:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 92805.000000 [06:26:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 125598.000000 [06:27:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79490.000000 [06:29:14] aliasd on nightshade is CRITICAL: Connection refused [06:31:14] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:31:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [06:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:33:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [06:34:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 211897.000000 [06:35:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [06:36:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:36:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:36:43] / on thyme is UNKNOWN: NRPE: Unable to read output [06:36:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [06:37:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81803.000000 [06:37:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [06:38:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [06:38:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [06:38:44] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:39:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [06:39:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [06:39:33] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:41:13] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:43:54] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21754.000000 [06:44:23] aliasd on yarrow is CRITICAL: Connection refused [06:45:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52410.000000 [06:51:43] Load avg. on daphne is WARNING: WARNING - load average: 15.14, 15.39, 11.80 [06:52:43] Load avg. on daphne is OK: OK - load average: 13.72, 14.92, 11.86 [06:53:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81639 MB (13% inode=99%): [06:54:15] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:57:15] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:00:44] Load avg. on daphne is WARNING: WARNING - load average: 19.88, 17.72, 14.39 [07:01:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:07:13] MySQL on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [07:07:22] MySQL slave on z-dat-s3-a is CRITICAL: Cant connect to MySQL server on z-dat-s3-a (146) [07:13:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:20:14] MySQL on z-dat-s3-a is OK: Uptime: 1136 Threads: 17 Questions: 3599 Slow queries: 0 Opens: 279 Flush tables: 1 Open tables: 268 Queries per second avg: 3.168 [07:20:23] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2510 [07:20:55] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:20:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:21:34] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:22:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14499.000000 [07:23:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:23:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473561 MB (8% inode=64%): [07:23:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:26:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 77966 [07:27:05] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 129259.000000 [07:27:05] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103278.000000 [07:27:05] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 96466.000000 [07:27:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83091.000000 [07:29:14] aliasd on nightshade is CRITICAL: Connection refused [07:31:14] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:31:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [07:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:33:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [07:35:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 215556.000000 [07:36:13] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:36:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:36:22] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [07:36:44] / on thyme is UNKNOWN: NRPE: Unable to read output [07:36:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [07:37:05] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77912.000000 [07:37:55] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [07:38:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [07:38:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [07:39:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [07:39:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [07:39:33] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:39:43] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:41:14] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:43:56] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25355.000000 [07:44:24] aliasd on yarrow is CRITICAL: Connection refused [07:45:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 56010.000000 [07:46:43] Load avg. on daphne is OK: OK - load average: 9.61, 11.70, 14.80 [07:53:23] MySQL slave on z-dat-s3-a is OK: Uptime: 3122 Threads: 18 Questions: 1069060 Slow queries: 93 Opens: 12218 Flush tables: 1 Open tables: 12172 Queries per second avg: 342.427 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1659 [07:54:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81429 MB (13% inode=99%): [07:55:13] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [07:56:44] Load avg. on daphne is WARNING: WARNING - load average: 18.76, 16.57, 15.64 [07:57:14] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:01:14] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:09:24] aliasd on yarrow is OK: TCP OK - 0.006 second response time on port 984 [500 Not found.] [08:11:43] Load avg. on daphne is OK: OK - load average: 10.06, 13.68, 14.93 [08:14:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:17:14] aliasd on nightshade is OK: TCP OK - 0.010 second response time on port 984 [500 Not found.] [08:20:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:20:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:21:33] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:22:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18098.000000 [08:23:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:23:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473367 MB (8% inode=64%): [08:24:23] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:27:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86690.000000 [08:27:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 74299 [08:28:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 132919.000000 [08:28:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106939.000000 [08:28:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100125.000000 [08:31:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:32:14] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:32:33] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [08:33:15] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [08:35:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 219156.000000 [08:36:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:36:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:36:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [08:36:44] Load avg. on nightshade is WARNING: WARNING - load average: 14.93, 16.85, 15.18 [08:36:44] / on thyme is UNKNOWN: NRPE: Unable to read output [08:36:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [08:37:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [08:38:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73128.000000 [08:39:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [08:39:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [08:39:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [08:39:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [08:39:43] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:39:43] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:39:43] Load avg. on nightshade is OK: OK - load average: 13.14, 14.74, 14.62 [08:41:15] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:44:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29015.000000 [08:45:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59610.000000 [08:54:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81218 MB (13% inode=99%): [08:55:14] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [08:58:15] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:02:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:06:43] Load avg. on nightshade is WARNING: WARNING - load average: 15.88, 16.25, 15.10 [09:15:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:21:33] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:21:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:43] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 34.10, 20.79, 17.83 [09:23:43] Load avg. on nightshade is WARNING: WARNING - load average: 24.97, 20.76, 18.02 [09:23:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21759.000000 [09:24:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:24:13] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 469613 MB (8% inode=64%): [09:24:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 90352.000000 [09:28:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 68227 [09:29:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 136579.000000 [09:29:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110598.000000 [09:29:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103786.000000 [09:32:14] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:32:43] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:32:43] Load avg. on nightshade is OK: OK - load average: 13.31, 12.80, 14.98 [09:34:13] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [09:35:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 222756.000000 [09:36:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [09:36:43] / on thyme is UNKNOWN: NRPE: Unable to read output [09:36:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [09:37:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:37:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:38:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67004.000000 [09:38:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [09:40:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [09:40:02] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [09:40:02] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [09:40:13] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [09:40:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:40:44] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:42:14] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:44:54] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32615.000000 [09:45:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63211.000000 [09:54:15] MySQL on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [09:54:23] MySQL slave on z-dat-s6-a is CRITICAL: Cant connect to MySQL server on z-dat-s6-a (146) [09:54:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80970 MB (13% inode=99%): [09:55:14] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:57:14] MySQL on z-dat-s6-a is OK: Uptime: 531 Threads: 9 Questions: 3032 Slow queries: 4 Opens: 240 Flush tables: 1 Open tables: 100 Queries per second avg: 5.709 [09:57:23] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1910 [09:58:15] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [09:59:15] aliasd on nightshade is CRITICAL: Connection refused [09:59:24] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2137 [09:59:24] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2211 [09:59:24] MySQL slave on z-dat-s6-a is OK: Uptime: 664 Threads: 15 Questions: 55722 Slow queries: 13 Opens: 949 Flush tables: 1 Open tables: 175 Queries per second avg: 83.918 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1720 [10:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:02:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:15:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:21:34] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:21:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:21:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:24] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3615 [10:23:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25359.000000 [10:24:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:24:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 471045 MB (8% inode=64%): [10:24:23] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3606 [10:24:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:25:16] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2180 [10:28:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 93951.000000 [10:28:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64196 [10:29:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 140179.000000 [10:29:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114199.000000 [10:29:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 107391.000000 [10:32:15] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:32:43] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [10:34:15] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [10:36:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 226416.000000 [10:36:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [10:36:44] / on thyme is UNKNOWN: NRPE: Unable to read output [10:36:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [10:37:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:37:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:38:56] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [10:39:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63426.000000 [10:40:02] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [10:40:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [10:40:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [10:40:14] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [10:40:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:40:44] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:42:14] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:45:53] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36275.000000 [10:46:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66871.000000 [10:50:13] MySQL slave on z-dat-s4-a is OK: Uptime: 6867472 Threads: 1 Questions: 613315828 Slow queries: 353 Opens: 689 Flush tables: 11 Open tables: 229 Queries per second avg: 89.307 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1741 [10:54:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80635 MB (13% inode=99%): [10:55:14] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:58:15] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:59:14] aliasd on nightshade is CRITICAL: Connection refused [11:03:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:15:23] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3514 [11:15:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:21:33] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:21:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:21:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28958.000000 [11:24:23] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5090 [11:24:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:25:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:25:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 470714 MB (8% inode=64%): [11:28:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 97552.000000 [11:28:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61029 [11:29:04] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 143779.000000 [11:29:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117799.000000 [11:29:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110991.000000 [11:30:23] MySQL slave on z-dat-s7-a is OK: Uptime: 84102 Threads: 5 Questions: 26133514 Slow queries: 3081 Opens: 78549 Flush tables: 1 Open tables: 3085 Queries per second avg: 310.735 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1763 [11:31:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:32:15] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:33:44] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [11:35:15] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [11:36:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 230017.000000 [11:36:23] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3419 [11:37:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:37:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:37:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [11:37:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [11:37:43] / on thyme is UNKNOWN: NRPE: Unable to read output [11:39:53] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [11:40:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [11:40:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [11:40:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60420.000000 [11:40:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [11:40:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [11:40:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:40:44] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:42:17] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:44:23] MySQL slave on z-dat-s3-a is OK: Uptime: 16983 Threads: 16 Questions: 9240202 Slow queries: 19068 Opens: 86630 Flush tables: 1 Open tables: 16384 Queries per second avg: 544.85 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1775 [11:45:54] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39875.000000 [11:46:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70470.000000 [11:55:15] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:55:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80343 MB (13% inode=99%): [11:58:15] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:58:29] @replag [11:59:16] aliasd on nightshade is CRITICAL: Connection refused [12:04:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:15:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:21:33] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:21:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32619.000000 [12:25:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:25:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 468137 MB (8% inode=64%): [12:25:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:26:13] /tmp on willow is WARNING: DISK WARNING - free space: / 14120 MB (13% inode=98%): [12:26:54] / on willow is WARNING: DISK WARNING - free space: / 13219 MB (12% inode=98%): [12:28:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 101152.000000 [12:28:55] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 58369 [12:28:55] / on willow is CRITICAL: DISK CRITICAL - free space: / 10751 MB (10% inode=97%): [12:29:14] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 10460 MB (9% inode=97%): [12:30:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 147439.000000 [12:30:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121459.000000 [12:30:14] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114651.000000 [12:32:15] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:34:43] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [12:35:14] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [12:36:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 233617.000000 [12:37:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [12:37:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [12:37:44] / on thyme is UNKNOWN: NRPE: Unable to read output [12:38:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [12:38:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:39:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [12:40:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [12:40:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [12:40:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58052.000000 [12:40:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [12:40:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [12:40:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:40:44] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:43:14] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:44:14] aliasd on nightshade is OK: TCP OK - 0.004 second response time on port 984 [500 Not found.] [12:45:54] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43476.000000 [12:47:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73971.000000 [12:49:54] / on willow is OK: DISK OK - free space: / 35737 MB (34% inode=99%): [12:50:15] /tmp on willow is OK: DISK OK - free space: / 35737 MB (34% inode=99%): [12:56:15] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:56:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 80054 MB (13% inode=99%): [12:59:14] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:05:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:10:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 32006 MB (3% inode=99%): [13:16:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:22:33] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:22:52] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:22:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:25:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21601.000000 [13:26:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:26:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 493338 MB (9% inode=65%): [13:29:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103289.000000 [13:29:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57245 [13:31:04] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 150682.000000 [13:31:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122377.000000 [13:31:14] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117129.000000 [13:33:13] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:34:43] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [13:35:15] NTP on hemlock is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [13:36:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 235256.000000 [13:37:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [13:38:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:38:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:38:45] / on thyme is UNKNOWN: NRPE: Unable to read output [13:38:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [13:39:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [13:40:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [13:40:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [13:40:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57117.000000 [13:40:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [13:40:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:40:44] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:41:16] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [13:43:15] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:45:55] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34911.000000 [13:47:55] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72672.000000 [13:48:15] NTP on hemlock is OK: NTP OK: Offset -0.006378 secs [13:56:17] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [13:56:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79746 MB (13% inode=99%): [13:59:16] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:01:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:05:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:06:00] /sql on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:11:55] /tmp on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:11:55] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:11:55] / on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:11:55] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:11:55] /sql on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:11:56] / on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:11:56] /tmp on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:11:57] /tmp on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:11:57] / on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:11:58] / on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:04] SSH on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:04] SSH on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:04] /tmp on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:15] Load avg. on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:15] Load avg. on z-dat-s7-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:15] Load avg. on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:15] SSH on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:15] /tmp on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:16] Load avg. on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:16] Load avg. on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:17] RAID on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:24] Environment IPMI on hyacinth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:43] SSH on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:43] SMTP on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:54] / on z-dat-s4-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [14:12:54] SSH on z-dat-s6-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:03] SMTP on hyacinth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:33] SMTP on z-dat-s4-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:33] SMTP on z-dat-s3-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:45] SMTP on z-dat-s7-a is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:03] MySQL on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [14:14:03] MySQL on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [14:14:15] MySQL on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [14:14:15] MySQL slave on z-dat-s4-a is CRITICAL: (Service Check Timed Out) [14:14:15] MySQL on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [14:14:23] MySQL slave on z-dat-s6-a is CRITICAL: (Service Check Timed Out) [14:14:23] MySQL slave on z-dat-s7-a is CRITICAL: (Service Check Timed Out) [14:14:23] MySQL slave on z-dat-s3-a is CRITICAL: (Service Check Timed Out) [14:16:45] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:19:33] SMTP on z-dat-s6-a is OK: SMTP OK - 0.307 sec. response time [14:19:34] SMTP on z-dat-s7-a is OK: SMTP OK - 0.598 sec. response time [14:19:44] SSH on z-dat-s4-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:19:45] / on z-dat-s4-a is OK: DISK OK - free space: / 6649 MB (22% inode=84%): [14:19:45] SSH on z-dat-s6-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:19:54] /tmp on hyacinth is OK: DISK OK - free space: /tmp 32946 MB (99% inode=97%): [14:19:54] /sql on z-dat-s3-a is OK: DISK OK - free space: /sql 285648 MB (29% inode=99%): [14:19:54] / on z-dat-s7-a is OK: DISK OK - free space: / 6649 MB (22% inode=84%): [14:19:54] /sql on z-dat-s4-a is OK: DISK OK - free space: /sql 86152 MB (14% inode=99%): [14:19:54] /sql on z-dat-s6-a is OK: DISK OK - free space: /sql 370187 MB (45% inode=99%): [14:19:54] /tmp on z-dat-s3-a is OK: DISK OK - free space: /tmp 32919 MB (99% inode=97%): [14:19:54] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 40406 MB (7% inode=99%): [14:19:55] / on hyacinth is OK: DISK OK - free space: / 6649 MB (22% inode=84%): [14:19:55] /tmp on z-dat-s4-a is OK: DISK OK - free space: /tmp 36466 MB (99% inode=97%): [14:19:56] / on z-dat-s3-a is OK: DISK OK - free space: / 6649 MB (22% inode=84%): [14:19:56] SSH on z-dat-s3-a is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:19:57] SSH on hyacinth is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [14:20:15] Load avg. on z-dat-s4-a is OK: OK - load average: 0.96, 0.27, 0.45 [14:20:15] Load avg. on z-dat-s7-a is OK: OK - load average: 0.96, 0.27, 0.45 [14:20:15] /tmp on z-dat-s6-a is OK: DISK OK - free space: /tmp 47273 MB (99% inode=97%): [14:20:15] Load avg. on z-dat-s3-a is OK: OK - load average: 0.96, 0.27, 0.45 [14:20:15] Load avg. on z-dat-s6-a is OK: OK - load average: 0.96, 0.27, 0.45 [14:20:16] Load avg. on hyacinth is OK: OK - load average: 1.08, 0.30, 0.46 [14:20:16] RAID on hyacinth is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [14:20:24] SMTP on z-dat-s4-a is OK: SMTP OK - 0.038 sec. response time [14:20:24] SMTP on z-dat-s3-a is OK: SMTP OK - 0.007 sec. response time [14:20:33] Environment IPMI on hyacinth is OK: ok: temperature ok fan ok voltage ok chassis ok [14:22:32] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:22:53] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:53] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:54] MySQL on z-dat-s4-a is OK: Uptime: 360 Threads: 1 Questions: 37819 Slow queries: 0 Opens: 53 Flush tables: 1 Open tables: 43 Queries per second avg: 105.52 [14:25:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4710.000000 [14:26:03] MySQL slave on z-dat-s4-a is OK: Uptime: 370 Threads: 1 Questions: 50670 Slow queries: 0 Opens: 58 Flush tables: 1 Open tables: 48 Queries per second avg: 136.945 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1730 [14:26:14] MySQL on z-dat-s6-a is OK: Uptime: 380 Threads: 6 Questions: 5925 Slow queries: 2 Opens: 107 Flush tables: 1 Open tables: 100 Queries per second avg: 15.592 [14:26:14] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2068 [14:26:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:27:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 492951 MB (9% inode=65%): [14:28:16] MySQL slave on z-dat-s6-a is OK: Uptime: 500 Threads: 3 Questions: 52277 Slow queries: 7 Opens: 164 Flush tables: 1 Open tables: 155 Queries per second avg: 104.554 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1709 [14:29:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104494.000000 [14:29:53] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3536.000000 [14:29:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 55684 [14:31:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 151622.000000 [14:31:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120270.000000 [14:31:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s7-a (146) [14:33:15] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:34:44] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [14:36:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 236183.000000 [14:38:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:38:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:38:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [14:38:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [14:38:45] / on thyme is UNKNOWN: NRPE: Unable to read output [14:38:57] hi [14:39:00] anyone there? [14:39:52] need some help on the toolserver [14:39:54] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1725.000000 [14:39:54] / on willow is WARNING: DISK WARNING - free space: / 15839 MB (15% inode=98%): [14:40:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [14:40:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [14:40:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54974.000000 [14:40:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [14:40:14] /tmp on willow is WARNING: DISK WARNING - free space: / 15562 MB (14% inode=98%): [14:40:38] i can't seem to connect to the database [14:40:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:40:44] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:40:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [14:41:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [14:41:25] for example when i connect with python i get this error: "Lost connection to MySQL server at 'reading initial communication packet', system error: 0", None [14:42:05] have just been following the instructions on https://wiki.toolserver.org/view/Database_access [14:42:16] and tried the example scripts, but with no luck [14:42:44] MySQL on z-dat-s3-a is OK: Uptime: 1369 Threads: 5 Questions: 1405 Slow queries: 0 Opens: 76 Flush tables: 1 Open tables: 55 Queries per second avg: 1.26 [14:43:14] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). warnings detected. run with -v for information. [14:43:32] MySQL on z-dat-s7-a is OK: Uptime: 1419 Threads: 3 Questions: 214 Slow queries: 1 Opens: 40 Flush tables: 1 Open tables: 32 Queries per second avg: 0.150 [14:43:44] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3108 [14:44:16] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 11507 MB (10% inode=98%): [14:44:54] / on willow is CRITICAL: DISK CRITICAL - free space: / 10706 MB (10% inode=97%): [14:45:54] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16286.000000 [14:48:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71765.000000 [14:49:44] MySQL slave on z-dat-s7-a is OK: Uptime: 1789 Threads: 8 Questions: 234282 Slow queries: 5 Opens: 484 Flush tables: 1 Open tables: 471 Queries per second avg: 130.956 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1758 [14:55:03] anyone there? [14:56:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79416 MB (13% inode=99%): [14:57:04] edin: it seems you picked a bad time :/ [14:57:11] might be best to send an email to the mailing list [14:57:13] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:57:17] hi lydia [14:57:21] yes, probably [14:57:25] :) [14:57:35] i now suddenly see it actually works again :) [14:57:40] hah [14:57:42] even better of course [14:57:56] thanks anyway [14:58:00] sure [14:58:20] by the way, since i'm already here [14:58:47] and you do organize a lot on wikidata stuff [14:58:52] heh [14:58:53] yes... [14:59:15] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [14:59:20] i was wondering what the best thing would be to do when i want to automatically create a larger amount of pages on wikipedia [14:59:39] the thing is, i planned that before wikidata was implemented on wikipedia [14:59:52] depends how wikidata plays into it [15:00:03] what do you have in mind? [15:00:08] the generated pages would all have some interwiki links in it, but now, that is of course not necessary [15:00:16] yes [15:00:29] well, the plan was to generate all the NGC objects for the bosnian wikipedia [15:00:43] which are about 10-15k of pages [15:00:55] it's actually not good to have them in the article anymore unless it is done on purpose to overwrite the wikidata ones for some reason [15:01:14] because since they overwrite what comes from wikidata there is a good chance they will get outdated [15:01:21] what is the best thing to do if i dont want to put the interwiki links directly onto the pages [15:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:01:45] you add them via the api to the item for them on wikidata if they exist already [15:01:46] yes i know, thats also why i wanted to ask your thoughts on it [15:02:19] if you want to know how to use the api there are a few people in #wikimedia-wikidata who can help you better with the details [15:02:21] i would preferably need a bot flag on wikidata to do add the bs link on all the relevant ngc items [15:02:27] yes [15:02:34] there is a page on wikidata.org to request it [15:02:49] i know how to use the api for wikipedia; haven't tried wikidata though.. but that shouldnt be a problem [15:02:56] ok yeah [15:03:01] should be easy enough for you then [15:03:32] are there any specific requirements for getting this flag? [15:03:56] http://www.wikidata.org/wiki/Wikidata:Bots has details [15:04:06] or is it the same procedure as usual? [15:04:17] alright, thanks lydia [15:04:22] no problem [15:05:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:06:44] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3458 [15:08:15] Load avg. on ptolemy is WARNING: WARNING - load average: 17.96, 15.88, 11.05 [15:09:15] Load avg. on ptolemy is OK: OK - load average: 13.40, 14.99, 11.04 [15:16:15] aliasd on nightshade is CRITICAL: Connection refused [15:16:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:20:44] MySQL slave on z-dat-s3-a is OK: Uptime: 3648 Threads: 23 Questions: 2646626 Slow queries: 47 Opens: 23670 Flush tables: 1 Open tables: 15738 Queries per second avg: 725.500 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1679 [15:22:34] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:22:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:22:55] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:53] / on willow is OK: DISK OK - free space: / 51195 MB (48% inode=99%): [15:25:14] /tmp on willow is OK: DISK OK - free space: / 51033 MB (48% inode=99%): [15:25:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:27:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:28:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 452586 MB (8% inode=64%): [15:29:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106861.000000 [15:29:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52952 [15:31:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123870.000000 [15:32:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 151986.000000 [15:32:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 128982.000000 [15:34:16] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:35:43] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [15:37:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 234930.000000 [15:38:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [15:38:44] / on thyme is UNKNOWN: NRPE: Unable to read output [15:39:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:39:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:39:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [15:40:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [15:40:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [15:40:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52268.000000 [15:40:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:40:44] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:40:54] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [15:41:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [15:41:16] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [15:43:15] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:45:55] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3364.000000 [15:48:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70277.000000 [15:57:14] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [15:57:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 79125 MB (12% inode=99%): [16:00:14] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:04:54] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1691.000000 [16:05:16] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:16:16] aliasd on nightshade is CRITICAL: Connection refused [16:16:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:22:54] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:22:54] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:33] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:25:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:27:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:29:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 437660 MB (8% inode=64%): [16:29:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108582.000000 [16:29:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48244 [16:32:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 154187.000000 [16:32:16] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127530.000000 [16:32:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 132582.000000 [16:35:14] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:35:44] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [16:37:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 233484.000000 [16:39:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:39:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:39:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [16:39:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [16:39:44] / on thyme is UNKNOWN: NRPE: Unable to read output [16:40:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [16:40:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [16:40:03] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47449.000000 [16:40:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:40:44] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:41:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [16:41:03] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [16:42:03] / on willow is WARNING: DISK WARNING - free space: / 16371 MB (15% inode=98%): [16:42:14] /tmp on willow is WARNING: DISK WARNING - free space: / 16172 MB (15% inode=98%): [16:42:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [16:43:14] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:48:04] / on willow is CRITICAL: DISK CRITICAL - free space: / 11248 MB (10% inode=98%): [16:48:14] /tmp on willow is CRITICAL: DISK CRITICAL - free space: / 10957 MB (10% inode=98%): [16:49:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68776.000000 [16:57:14] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [16:57:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78897 MB (12% inode=99%): [17:00:14] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:01:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:06:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:10:03] / on willow is OK: DISK OK - free space: / 36379 MB (34% inode=99%): [17:10:15] /tmp on willow is OK: DISK OK - free space: / 36379 MB (34% inode=99%): [17:16:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:17:14] aliasd on nightshade is CRITICAL: Connection refused [17:23:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:05] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:34] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:25:34] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:29:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110734.000000 [17:30:14] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 465943 MB (8% inode=64%): [17:30:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44704 [17:32:04] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 154938.000000 [17:32:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 136182.000000 [17:33:14] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 131190.000000 [17:35:15] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:35:43] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [17:38:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 232202.000000 [17:39:23] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [17:39:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [17:39:43] / on thyme is UNKNOWN: NRPE: Unable to read output [17:40:03] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [17:40:03] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [17:40:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44456.000000 [17:40:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:40:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:41:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [17:41:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [17:41:43] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:41:43] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:42:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [17:43:15] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:49:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66188.000000 [17:57:14] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [17:57:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78658 MB (12% inode=99%): [18:00:13] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:06:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:17:15] aliasd on nightshade is CRITICAL: Connection refused [18:17:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:23:34] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:24:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:15] aliasd on nightshade is OK: TCP OK - 0.003 second response time on port 984 [500 Not found.] [18:25:34] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:16] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:29:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 112796.000000 [18:30:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 457494 MB (8% inode=64%): [18:30:51] hello [18:30:54] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 41575 [18:30:55] maintenance tonight [18:31:13] turnera. damiana, wolfsbane and ortelius will be renumbered [18:31:31] i am going to prepare [18:31:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:31:50] official maintenance window is starting in about half an hour [18:33:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 155543.000000 [18:33:43] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 139841.000000 [18:34:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134849.000000 [18:36:14] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:36:43] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [18:38:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 231997.000000 [18:39:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [18:39:44] RAID on thyme is UNKNOWN: NRPE: Unable to read output [18:39:45] / on thyme is UNKNOWN: NRPE: Unable to read output [18:41:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [18:41:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [18:41:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [18:41:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40992.000000 [18:41:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [18:41:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:41:15] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:41:43] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:41:43] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:43:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [18:44:14] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:50:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64553.000000 [18:57:15] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [18:57:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78444 MB (12% inode=99%): [19:00:14] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:06:15] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:18:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:24:04] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:04] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:24:33] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:25:11] i am still preparin [19:26:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:29:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114642.000000 [19:30:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 449850 MB (8% inode=64%): [19:32:14] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37449 [19:32:14] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:33:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 157438.000000 [19:33:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 143443.000000 [19:34:13] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138450.000000 [19:37:00] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [19:37:15] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:38:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 230561.000000 [19:39:25] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [19:40:43] / on thyme is UNKNOWN: NRPE: Unable to read output [19:40:43] RAID on thyme is UNKNOWN: NRPE: Unable to read output [19:41:44] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:41:44] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:42:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [19:42:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:42:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [19:42:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36579.000000 [19:42:04] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [19:42:18] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:42:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:43:15] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [19:44:17] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:50:54] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63261.000000 [19:57:17] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [19:57:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 78162 MB (12% inode=99%): [20:00:15] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:07:18] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:19:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:24:32] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:25:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:25:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:27:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:28:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:29:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117038.000000 [20:30:15] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 425692 MB (7% inode=64%): [20:31:54] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:32:27] ok preparation is through [20:32:38] now i will reboot wolfsbane & ortelius [20:32:47] web stuff will be completely offline [20:33:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 159647.000000 [20:33:15] after that ill turn off damiana and reboot turnera to see if everything comes back up [20:33:42] then ill start damiana and after that manually turn on all services [20:33:44] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 147042.000000 [20:33:54] this will cause interruptions for web and database stuff [20:34:00] even auth should not work [20:34:15] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 142050.000000 [20:34:17] ok rebooting the web servers now [20:35:03] ah and dns is then broken too [20:35:10] hope this does not take too long :D [20:36:44] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [20:37:15] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [20:38:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 228915.000000 [20:39:24] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [20:40:44] / on thyme is UNKNOWN: NRPE: Unable to read output [20:40:45] RAID on thyme is UNKNOWN: NRPE: Unable to read output [20:41:45] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:41:45] APT on yarrow is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:42:04] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [20:42:04] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [20:42:04] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [20:42:04] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36538.000000 [20:42:05] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [20:42:15] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:42:16] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:43:16] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [20:44:15] APT on mayapple is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:44:57] web services appear to be down [20:45:11] true [20:45:31] ill now break everything :D [20:45:40] lol nosy [20:45:42] so damiana/turnera are next [20:47:45] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:15:53] nosy: I'm unable to make ssh connection with toolserver [21:16:08] Krinkle: its really broken please wait [21:16:10] both login.toolserver.org (willow?) and nightshade.toolserver.org ask for a password [21:16:29] and nfs does not work thats why your key does not too [21:41:03] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:44:03] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:44:45] MySQL on ha-sql.esi is CRITICAL: Cant connect to MySQL server on ha-sql.esi (146) [21:45:13] /tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:13] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:13] Sensors on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:13] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:13] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:14] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:14] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:15] /home on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:15] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:45:48] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3441.000000 [21:45:48] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3442.000000 [21:45:48] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:48] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:48] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:49] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [21:45:52] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [21:45:52] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2464.000000 [21:45:52] SMTP on daphne is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:52] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:52] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:53] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:53] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:45:54] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:01] DiskSuite on turnera is CRITICAL: CRITICAL - submirror d11 of mirror d10 is Resyncing and submirror d12 of mirror d10 is Resyncing [21:46:02] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [21:46:02] Authoritative DNS on ha-dns-auth is CRITICAL: CRITICAL - Plugin timed out while executing system call [21:46:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:02] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:02] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:03] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:11] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 82.02, 77.09, 71.02 [21:46:12] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:12] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:12] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:12] SRaid on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:12] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:13] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:13] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:21] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3475.000000 [21:46:21] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:22] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:22] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:22] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2345 [21:46:22] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:22] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:23] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:23] / on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:31] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:46:32] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:42] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:42] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:42] Environment IPMI on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:42] MySQL on ha-sql.esi is OK: Uptime: 4 Threads: 1 Questions: 1 Slow queries: 0 Opens: 15 Flush tables: 1 Open tables: 8 Queries per second avg: 0.250 [21:46:52] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:52] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:52] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [21:46:52] Load avg. on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:46:52] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:47:01] DiskSuite on turnera is OK: OK - No disk failures detected [21:47:02] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:02] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:48:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3621.000000 [21:48:42] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3622.000000 [21:48:52] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1700.000000 [21:49:21] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [21:49:22] MySQL slave on daphne is OK: Uptime: 137143 Threads: 27 Questions: 77863523 Slow queries: 5810 Opens: 3825 Flush tables: 1 Open tables: 1298 Queries per second avg: 567.754 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1632 [21:49:51] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1980 [21:50:52] MySQL slave on z-dat-s3-a is OK: Uptime: 27063 Threads: 4 Questions: 22645256 Slow queries: 1106 Opens: 132951 Flush tables: 1 Open tables: 16384 Queries per second avg: 836.760 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1789 [21:52:42] SMTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:45] nosy: around? [21:52:56] Danny_B|webgate: still maintenance [21:52:57] yes [21:53:02] its not working :D [21:53:05] i guess [21:53:23] nosy: NFS server ha-nfs.esi not responding still trying <- known issue? [21:55:21] SSH on willow is CRITICAL: Server answer: [21:57:45] /home on hemlock is OK: DISK OK - free space: /home 16988 MB (33% inode=85%): [22:00:33] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:01:23] SMTP on willow is WARNING: recv() failed [22:02:33] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:42] / on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:06:42] /tmp on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:06:52] Environment IPMI on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:07:01] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [22:13:12] /home on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:22:22] hmm, ping toolserver.orgPING toolserver.org (91.198.174.214) <-- some trouble with dns ? [22:30:55] phe: i already have 185.... . but i just checked all secondary nameserver and they all have the new record. [22:32:42] my brower timeout on http://toolserver.org/ and ping give me 91.198.174.214 for toolserver, ping give me 185.15.59.201 for nightshade.toolserver.org [22:33:01] and I can't ssh on yarrow or nightshade [22:33:33] you cannot ssh because ldap runs on the ha-cluster which is not available [22:33:35] I'm also seeing problems [22:33:52] oh, right, the ha-cluster [22:35:12] good night [22:35:21] I wonder why I don't see toolserver.org in the same ip block as nightshade.toolserver.org, something wrong on my side ? [22:35:33] Platonides: the operator account is not in ldap but our home is on ha cluster, so we cannot use it, too [22:35:57] * phe scrollback and read [22:36:03] nightshade/yarrow were changed some days ago [22:37:16] yeps, I didn't notice web offline was annouced a bit back and I thought all ip was changed [22:37:45] SMTP on amaranth is OK: SMTP OK - 7.798 sec. response time [22:39:42] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [22:39:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77905 MB (12% inode=99%): [22:39:43] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69090.000000 [22:39:51] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [22:39:52] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:02] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:02] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 124454.000000 [22:40:02] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [22:40:02] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:40:22] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:40:42] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [22:40:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [22:40:42] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 405286 MB (7% inode=64%): [22:40:42] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [22:40:52] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [22:40:52] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:40:52] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:41:11] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [22:41:11] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43683.000000 [22:41:21] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:41:21] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 166401.000000 [22:41:21] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 235426.000000 [22:41:22] RAID on thyme is UNKNOWN: NRPE: Unable to read output [22:41:31] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:41:32] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:41:32] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [22:41:32] / on thyme is UNKNOWN: NRPE: Unable to read output [22:41:41] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:41:42] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [22:44:16] any eta on getting willow back up? [22:44:42] SMTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:45:01] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:22] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:22] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:22] Sensors on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:22] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:22] /tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:22] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:23] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:23] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:45:52] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [22:45:52] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:01] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:02] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:02] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:02] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:02] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [22:46:02] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:03] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:03] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:04] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:04] SMTP on daphne is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:11] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 191.59, 184.28, 165.06 [22:46:11] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:12] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:12] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:12] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:21] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:46:21] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:21] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:21] SRaid on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:22] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:22] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:22] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:23] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:23] / on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:24] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:31] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:32] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:32] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:42] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:43] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:43] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:43] Environment IPMI on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:43] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:43] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:43] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:46:44] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [22:46:52] Load avg. on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:46:52] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:47:01] Betacommand: server willow is running and online, only some services like ldap or filesystem are not available ;-) [22:47:02] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:47:02] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:47:42] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:42] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:42] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [22:47:56] Merlissimo: if willow is up, ssh access isnt [22:48:15] only ldap isn't [22:48:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7221.000000 [22:48:43] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7222.000000 [22:48:48] so key verification fails [22:49:22] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7256.000000 [22:49:54] Merlissimo: "Running" is saying a lot when uptimes gives back "-bash: fork: Not enough space" :) [22:49:59] uptime* [22:55:24] SSH on willow is CRITICAL: Server answer: [22:56:08] we should keep our fingers crossed for nosy [22:56:18] indeed :) [22:57:07] Merlissimo: and toes as well ;-) [22:58:27] Danny_B|webgate: too complicated. how should i write here if i have to cross fingers AND toes? [22:58:45] ;-) [22:58:58] Merlissimo: use your nose, bro... [23:01:24] * jem- tries to cross his eyes [23:01:32] Sun Grid Engine execd on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:03:31] SMTP on willow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:35] what's up? something wrong with the toolserver? [23:04:53] or is it something with my link here? [23:06:43] / on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:06:51] Environment IPMI on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:06:51] /tmp on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:07:01] Load avg. on willow is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [23:07:24] toolserver seems dead from here. [23:10:33] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:11:14] speckmade: its down for maintenance [23:12:03] SMTP on daphne is OK: SMTP OK - 7.551 sec. response time [23:12:03] SMTP on yarrow is OK: SMTP OK - 7.536 sec. response time [23:13:24] /home on hemlock is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:15:17] Siri. [23:15:27] legoktm: okay, thanks. [23:16:28] Can I expect to use it tomorrow? [23:16:49] it should be relatively soon [23:19:03] SMTP on daphne is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:19:03] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:31:23] SSH on z-dat-s2-b is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:39:13] Authoritative DNS on ha-dns-auth is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:39:45] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [23:39:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72691.000000 [23:39:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 77896 MB (12% inode=99%): [23:40:03] FMA on thyme is CRITICAL: ERROR - unexpected output from snmpwalk [23:40:03] SMTP on yarrow is OK: SMTP OK - 7.538 sec. response time [23:40:03] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:40:03] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:40:03] Load avg. on thyme is UNKNOWN: NRPE: Unable to read output [23:40:03] APT on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:40:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 128060.000000 [23:40:25] APT on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:40:46] /mnt on thyme is UNKNOWN: NRPE: Unable to read output [23:40:46] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [23:40:46] /tmp on thyme is UNKNOWN: NRPE: Unable to read output [23:40:46] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 405286 MB (7% inode=64%): [23:40:53] Environment IPMI on thyme is UNKNOWN: NRPE: Unable to read output [23:41:04] APT on sage is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:41:04] APT on nightshade is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [23:41:13] NTP on thyme is CRITICAL: NTP CRITICAL: Server not synchronized, Offset unknown [23:41:13] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47284.000000 [23:41:24] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 170001.000000 [23:41:24] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:25] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 239026.000000 [23:41:25] RAID on thyme is UNKNOWN: NRPE: Unable to read output [23:41:33] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:41:34] SMF on web.amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:41:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob: [23:41:34] / on thyme is UNKNOWN: NRPE: Unable to read output [23:41:46] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:41:46] APT on yucca is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [23:44:45] SMTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:45:03] aliasd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:24] Sun Grid Engine execd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:25] aliasd on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:25] Sensors on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:25] /tmp on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:25] /tmp on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:25] SRaid on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:25] Sensors on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:26] / on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:45:54] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [23:46:04] SMTP on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:46:04] FC 0/9 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:04] FC 0/20 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:04] FC 0/11 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:13] FMA on amaranth is CRITICAL: ERROR - unexpected output from snmpwalk [23:46:13] Load avg. on nightshade is CRITICAL: CRITICAL - load average: 301.16, 293.82, 275.06 [23:46:13] FC 0/14 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:13] FC 0/22 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:13] FC 0/13 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:14] FC 0/23 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:14] eth0 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:15] FC 0/12 [thyme] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:15] FC 0/21 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:16] NTP on amaranth is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:46:23] SMTP on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:46:23] FC 0/15 [fsw2-n1-oe16-esams] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:23] FC 0/3 [cassia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:24] SRaid on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:24] Environment IPMI on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:24] Sun Grid Engine execd on nightshade is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:24] FC 0/4 [hyacinth] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:25] FC 0/16 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:33] FC 0/5 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:34] FC 0/17 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:34] Load avg. on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:34] / on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:34] Load avg. on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:46] Environment IPMI on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:46] FC 0/6 [rosemary] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:46] FC 0/18 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:46] FC 0/0 [SAN far1-n1-oe16-esams A1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:46] FC 0/2 [hemlock] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:46] FC 0/8 [adenia] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:46] FC 0/10 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:46:47] MySQL slave on z-dat-s5-b is CRITICAL: (Return code of 139 is out of bounds) [23:46:54] SMF on amaranth is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:46:54] Load avg. on yarrow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:47:05] SMTP on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:47:05] Sun Grid Engine execd on mayapple is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:47:14] aliasd on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:47:45] FC 0/1 [far1-n1-oe16-esams B1] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:45] FC 0/7 [daphne] on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:47:45] FC 0/19 on fsw1-n1-oe16-esams.mgmt is UNKNOWN: ERROR: Description table : No response from remote host fsw1-n1-oe16-esams.mgmt. [23:48:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10822.000000 [23:48:45] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10823.000000 [23:49:24] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10856.000000 [23:51:04] SMTP on yarrow is OK: SMTP OK - 7.539 sec. response time [23:55:25] SSH on willow is CRITICAL: Server answer: