[00:04:13] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 62940 MB (6% inode=99%): [00:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:16:22] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:31:13] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [00:31:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 66041 MB (10% inode=99%): [00:32:53] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [00:36:13] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:38:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:38:43] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:38:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:39:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:42:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 117878 MB (2% inode=68%): [00:46:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:58:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 107788.000000 [01:07:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:16:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:22:32] Free Memory on damiana is WARNING: WARNING - 6.3% (526628 kB) free! [01:31:13] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [01:31:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65859 MB (10% inode=99%): [01:32:52] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [01:36:13] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:36:33] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (414880 kB) free! [01:38:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:38:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:38:43] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:38:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:39:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:42:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 118121 MB (2% inode=68%): [01:47:33] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:58:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105702.000000 [02:05:32] Free Memory on damiana is WARNING: WARNING - 5.9% (491836 kB) free! [02:07:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:11:32] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (421336 kB) free! [02:12:33] Free Memory on damiana is WARNING: WARNING - 5.1% (426596 kB) free! [02:16:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:31:13] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [02:31:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65946 MB (10% inode=99%): [02:32:54] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [02:36:13] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:38:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:38:43] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:38:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:39:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:40:32] Free Memory on damiana is CRITICAL: CRITICAL - 2.9% (244552 kB) free! [02:42:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 117802 MB (2% inode=68%): [02:46:14] hmmm [02:47:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:47:48] Free Memory on damiana is OK: OK - 92.0% (7709496 kB) free. [02:51:08] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2034.000000 [02:51:51] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2055 [02:53:17] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57531 MB (5% inode=99%): [02:56:18] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 60577 MB (6% inode=99%): [02:56:50] MySQL slave on z-dat-s5-b is OK: Uptime: 266435 Threads: 5 Questions: 301137365 Slow queries: 7700 Opens: 28651 Flush tables: 1 Open tables: 255 Queries per second avg: 1130.247 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1778 [02:57:09] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1671.000000 [02:58:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106538.000000 [03:07:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:12:41] Danny_B: It's not really wrong. It's just not intended to be used as color formatting, as I understand it. [03:17:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:31:51] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65495 MB (10% inode=99%): [03:31:59] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [03:33:00] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:36:18] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [03:38:18] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:38:51] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [03:38:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:39:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:40:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:42:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 117080 MB (2% inode=68%): [03:47:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:58:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108438.000000 [04:07:08] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:17:18] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:30:16] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7846&oldid=7835&rcid=21639 * 91.198.174.202 * (+1) (updated page) [04:32:12] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [04:32:50] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65235 MB (10% inode=99%): [04:33:08] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [04:36:19] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [04:38:27] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:38:50] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [04:39:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:39:08] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [04:40:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:42:18] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 116854 MB (2% inode=68%): [04:48:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:59:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109796.000000 [05:07:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:17:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:33:10] APT on yarrow is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:33:10] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [05:33:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 65049 MB (10% inode=99%): [05:36:19] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:38:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:38:57] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:39:09] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [05:39:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:40:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:42:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 116712 MB (2% inode=68%): [05:48:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:57:19] /sql on z-dat-s5-b is WARNING: DISK WARNING - free space: /sql 69292 MB (10% inode=99%): [06:00:50] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 111074.000000 [06:07:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:17:19] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:33:10] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [06:33:11] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [06:33:49] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64842 MB (10% inode=99%): [06:36:19] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [06:38:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:38:58] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:39:10] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [06:39:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:40:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:42:19] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 116495 MB (2% inode=68%): [06:48:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:49:10] Free Memory on turnera is WARNING: WARNING - 6.0% (499248 kB) free! [06:54:10] Free Memory on turnera is CRITICAL: CRITICAL - 4.2% (348612 kB) free! [06:58:12] Free Memory on turnera is OK: OK - 87.9% (7368136 kB) free. [07:00:11] FC 0/3 [cassia] on fsw2-n1-oe16-esams.mgmt is UNKNOWN: ERROR opening session: SNMPv3 support is unavailable (Required module Digest/SHA1.pm not found). [07:02:02] Free Memory on turnera is OK: OK - 85.3% (7150216 kB) free. [07:18:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:33:11] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [07:33:52] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [07:34:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64645 MB (10% inode=99%): [07:37:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:37:02] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [07:38:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:39:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:40:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:40:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:40:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:43:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 117529 MB (2% inode=68%): [07:49:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:01:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113483.000000 [08:18:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:32:12] /sql on z-dat-s5-b is CRITICAL: DISK CRITICAL - free space: /sql 38585 MB (5% inode=99%): [08:33:12] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [08:33:52] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [08:34:41] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 64433 MB (10% inode=99%): [08:37:02] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [08:37:22] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:38:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:39:21] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:40:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:40:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:40:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:43:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 117098 MB (2% inode=68%): [08:43:42] /sql on ptolemy is OK: DISK OK - free space: /sql 136192 MB (22% inode=99%): [08:49:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:50:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122956 MB (20% inode=99%): [09:01:31] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116294.000000 [09:18:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:33:12] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [09:33:52] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [09:37:02] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [09:38:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:39:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:40:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [09:40:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:40:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:43:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 111151 MB (2% inode=68%): [09:49:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:50:41] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122721 MB (20% inode=99%): [10:01:31] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118117.000000 [10:07:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:18:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:23:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58368 MB (5% inode=99%): [10:24:12] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 62630 MB (6% inode=99%): [10:33:12] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [10:33:52] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [10:37:01] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [10:38:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:39:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:40:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:40:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:40:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:43:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 82160 MB (1% inode=68%): [10:49:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:50:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122450 MB (20% inode=99%): [11:01:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119221.000000 [11:07:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:18:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:33:12] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [11:33:52] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [11:37:02] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [11:38:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:39:21] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:40:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:40:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:40:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:43:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 65014 MB (1% inode=68%): [11:49:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:50:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 122212 MB (20% inode=99%): [12:01:31] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118340.000000 [12:07:01] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:18:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:33:12] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [12:33:51] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [12:37:01] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [12:38:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:39:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:40:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:40:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:40:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:43:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 55260 MB (1% inode=68%): [12:49:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:50:41] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121966 MB (20% inode=99%): [13:01:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114378.000000 [13:07:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:18:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:33:11] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [13:33:52] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [13:37:02] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [13:38:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:39:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:40:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:40:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:40:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:43:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 46536 MB (0% inode=68%): [13:49:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:50:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121665 MB (20% inode=99%): [14:01:31] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109462.000000 [14:18:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:33:12] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [14:33:52] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [14:37:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:37:02] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [14:38:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:39:21] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:40:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:40:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:40:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:43:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 38173 MB (0% inode=68%): [14:49:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:50:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121370 MB (20% inode=99%): [15:01:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99958.000000 [15:18:12] /sql on z-dat-s5-b is OK: DISK OK - free space: /sql 305166 MB (32% inode=99%): [15:18:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:24:32] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3920.000000 [15:25:02] MySQL slave on z-dat-s5-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3886 [15:28:02] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3537 [15:28:31] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3463.000000 [15:33:12] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [15:33:51] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [15:36:37] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1674.000000 [15:37:02] MySQL slave on z-dat-s5-b is OK: Uptime: 1162 Threads: 4 Questions: 445045 Slow queries: 15 Opens: 320 Flush tables: 1 Open tables: 195 Queries per second avg: 382.999 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1428 [15:37:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:37:03] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [15:38:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:39:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:40:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:40:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:40:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:43:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 30309 MB (0% inode=68%): [15:49:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:50:41] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 121104 MB (19% inode=99%): [16:01:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84016.000000 [16:17:41] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 4.946 second response time [16:18:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:18:32] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.401 second response time [16:25:32] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 2.166 second response time [16:33:11] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [16:33:52] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [16:37:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:37:02] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [16:38:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:39:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:40:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:40:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:40:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:43:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 25396 MB (0% inode=68%): [16:49:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:50:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120874 MB (19% inode=99%): [17:01:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 74016.000000 [17:18:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:33:12] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:33:52] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [17:37:02] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:37:02] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [17:38:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:39:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:40:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:40:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:40:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:43:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 21256 MB (0% inode=68%): [17:49:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:50:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120656 MB (19% inode=99%): [17:53:46] Hello all [17:56:05] Hi DaBPunkt ! :) [18:01:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67972.000000 [18:18:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:31:09] Hmm, ehh, is the HTTP server down? or s2? [18:31:19] any of the HTTP ones? [18:31:28] hmm [18:31:32] no [18:33:12] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [18:33:52] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [18:37:03] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:37:03] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [18:38:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:39:22] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:40:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:40:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:40:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:43:06] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 17791 MB (0% inode=68%): [18:49:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:50:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120406 MB (19% inode=99%): [19:01:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62935.000000 [19:12:22] Free Memory on damiana is CRITICAL: CRITICAL - 3.9% (325256 kB) free! [19:18:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:20:22] Free Memory on damiana is WARNING: WARNING - 5.1% (426424 kB) free! [19:21:22] Free Memory on damiana is CRITICAL: CRITICAL - 4.4% (372860 kB) free! [19:32:49] Free Memory on damiana is OK: OK - 92.0% (7709880 kB) free. [19:33:35] hello there [19:34:01] APT on yarrow is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [19:34:11] APT on nightshade is CRITICAL: APT CRITICAL: 7 packages available for upgrade (7 critical updates). [19:37:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:37:12] APT on mayapple is CRITICAL: APT CRITICAL: 10 packages available for upgrade (10 critical updates). [19:37:57] looks lie an update-run is needed… [19:38:01] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 3.314 second response time [19:39:19] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:39:54] APT on yucca is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:40:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:40:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:41:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:43:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 15597 MB (0% inode=68%): [19:47:10] some people really need uncompressed dump ? it's eating all user-store space, and user-store is slow as hell since 10 hours [19:47:11] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 57988 MB (5% inode=99%): [19:49:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:50:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 120148 MB (19% inode=99%): [19:52:12] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 58624 MB (6% inode=99%): [19:56:11] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:56:11] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:56:12] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:59:54] APT on yucca is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:01:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60025.000000 [20:12:01] toolserver.org HTTP on ortelius is WARNING: HTTP WARNING: HTTP/1.1 200 OK - 239 bytes in 0.961 second response time [20:13:11] toolserver.org HTTP on ortelius is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:19:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:21:02] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.349 second response time [20:26:34] hello [20:37:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:39:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:41:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:41:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [20:41:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:43:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 13179 MB (0% inode=68%): [20:46:29] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2004.000000 [20:47:01] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2017 [20:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:51:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119910 MB (19% inode=99%): [20:59:29] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1759.000000 [21:00:02] MySQL slave on z-dat-s5-b is OK: Uptime: 20545 Threads: 7 Questions: 3188762 Slow queries: 308 Opens: 28255 Flush tables: 1 Open tables: 256 Queries per second avg: 155.208 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1665 [21:01:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42850.000000 [21:19:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:37:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:39:28] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:41:01] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:41:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:41:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:43:11] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 10897 MB (0% inode=68%): [21:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:52:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119641 MB (19% inode=99%): [22:02:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32573.000000 [22:19:02] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:39:29] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:41:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:41:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [22:41:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:44:10] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 8558 MB (0% inode=68%): [22:50:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:52:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119425 MB (19% inode=99%): [22:57:01] nacht ts [23:02:53] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32244.000000 [23:07:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:19:01] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:20:11] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 7.810 second response time [23:22:02] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.149 second response time [23:29:11] toolserver.org HTTP on ortelius is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - 239 bytes in 5.211 second response time [23:41:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:41:11] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:41:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:44:12] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 6677 MB (0% inode=68%): [23:48:13] some people really need uncompressed dump ? it's eating all user-store space, and user-store is slow as hell since 12 hours [23:50:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:52:53] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 119170 MB (19% inode=99%): [23:57:01] toolserver.org HTTP on ortelius is OK: HTTP OK: HTTP/1.1 200 OK - 239 bytes in 0.005 second response time