[00:22:17] Free Memory on damiana is CRITICAL: CRITICAL - 2.4% (197280 kB) free! [00:25:41] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:25:52] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [00:26:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:27:51] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3179 [00:27:52] Free Memory on damiana is OK: OK - 92.0% (7711700 kB) free. [00:29:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 54545 MB (5% inode=99%): [00:30:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:40:58] nacht ts [00:47:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 184185 MB (3% inode=67%): [00:47:57] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:47:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:47:58] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86982 MB (14% inode=99%): [00:49:57] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122642.000000 [00:50:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:50:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 100392.000000 [00:51:57] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3642 [00:54:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:14:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:26:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:30:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:47:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 184063 MB (3% inode=67%): [01:48:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:48:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84908 MB (13% inode=99%): [01:48:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:49:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 124396.000000 [01:50:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:50:02] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102498.000000 [01:52:56] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4846 [01:54:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:03:34] anyone home? [02:07:40] Hi. [02:14:30] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:17:34] hi susan [02:26:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:30:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:34:40] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Digital1210]]": spam) [02:34:52] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Digital1210]] with an expiry time of infinite (account creation disabled): spam) [02:48:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 181648 MB (3% inode=67%): [02:48:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:48:57] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86638 MB (14% inode=99%): [02:48:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:50:01] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103356.000000 [02:50:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126184.000000 [02:51:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:52:56] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5465 [02:55:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:14:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:16:44] [[Special:Log/newusers]] create 10 * Affordablesup * (New user account) [03:24:38] [[User talk:Affordablesup]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7849&rcid=21651 * Affordablesup * (+468) (Created page with "With the aim of introducing the game of surfing to more and more people all around the world, we are very proud of being one of the world’s leading companies in the field of pa...") [03:26:06] [[Special:Log/delete]] delete 10 * MZMcBride * (deleted "[[02User talk:Affordablesup10]]": spam) [03:26:19] [[Special:Log/block]] block 10 * MZMcBride * (blocked [[02User:Affordablesup10]] with an expiry time of infinite (account creation disabled): spam) [03:27:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:30:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:48:13] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 180722 MB (3% inode=67%): [03:49:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:49:56] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86418 MB (14% inode=99%): [03:49:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:51:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105851.000000 [03:51:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126921.000000 [03:52:02] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:52:56] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8111 [03:56:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:14:55] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:27:23] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:30:22] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7850&oldid=7846&rcid=21654 * 91.198.174.202 * (+0) (updated page) [04:31:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:48:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 180018 MB (3% inode=67%): [04:49:59] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86272 MB (14% inode=99%): [04:49:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:50:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:51:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 108547.000000 [04:52:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:52:56] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10430 [04:52:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127462.000000 [04:56:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:17:14] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (378704 kB) free! [05:27:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:27:44] Free Memory on turnera is OK: OK - 91.9% (7704972 kB) free. [05:32:00] Free Memory on turnera is OK: OK - 86.5% (7251208 kB) free. [05:32:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:49:00] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 180356 MB (3% inode=67%): [05:50:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:50:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 86088 MB (14% inode=99%): [05:50:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:52:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110550.000000 [05:52:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:53:10] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [05:53:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130176.000000 [05:53:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11584 [05:57:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:31:10] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:31:20] APT on yucca is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:20] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:32:01] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:32:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:49:00] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 180154 MB (3% inode=67%): [06:50:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:50:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85970 MB (14% inode=99%): [06:50:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:52:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 111171.000000 [06:52:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:53:10] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [06:53:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 132120.000000 [06:53:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12185 [06:57:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:31:10] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:31:20] APT on yucca is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:20] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:31:51] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:32:00] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:32:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:49:00] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 179947 MB (3% inode=67%): [07:50:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:50:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85816 MB (14% inode=99%): [07:50:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:52:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113009.000000 [07:52:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:53:10] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [07:53:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 132062.000000 [07:53:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13813 [07:57:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:31:11] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:31:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:31:20] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:31:20] APT on yucca is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:32:00] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:32:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:46:14] [[Special:Log/newusers]] create 10 * Desrencse * (New user account) [08:49:00] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 180763 MB (3% inode=67%): [08:50:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:50:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85595 MB (14% inode=99%): [08:50:51] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:52:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114568.000000 [08:52:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:53:10] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [08:53:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133376.000000 [08:53:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14273 [08:57:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:16:15] morning, Silke_WMDE [09:16:30] hi [09:19:49] * Silke_WMDE is in a meeting [09:31:10] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:31:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:31:20] APT on yucca is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:31:20] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:32:01] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:32:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:49:00] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 178866 MB (3% inode=67%): [09:50:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:50:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85366 MB (14% inode=99%): [09:50:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:52:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116155.000000 [09:52:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:53:10] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [09:53:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 135421.000000 [09:53:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15499 [09:57:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:31:10] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:31:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:31:20] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:31:20] APT on yucca is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:32:00] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:32:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:37:10] Load avg. on ptolemy is WARNING: WARNING - load average: 20.04, 17.59, 11.16 [10:39:11] Load avg. on ptolemy is OK: OK - load average: 6.04, 13.43, 10.41 [10:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:49:00] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 178429 MB (3% inode=67%): [10:50:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:50:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 85113 MB (13% inode=99%): [10:50:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:52:09] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116934.000000 [10:52:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:53:10] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [10:53:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 137850.000000 [10:53:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15301 [10:57:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:25:20] Free Memory on damiana is WARNING: WARNING - 5.9% (492036 kB) free! [11:31:11] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:31:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:31:20] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:31:21] APT on yucca is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:31:50] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:32:00] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:32:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:41:19] Free Memory on damiana is CRITICAL: CRITICAL - 3.6% (303548 kB) free! [11:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:45:40] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:46:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:47:50] Free Memory on damiana is OK: OK - 92.1% (7715612 kB) free. [11:49:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 178145 MB (3% inode=67%): [11:50:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [11:50:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84878 MB (13% inode=99%): [11:52:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118693.000000 [11:52:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:53:12] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [11:53:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 140988.000000 [11:53:55] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16821 [11:57:30] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:57:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:29:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 53654 MB (5% inode=99%): [12:31:12] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:31:55] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:31:55] APT on yucca is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:31:55] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:32:12] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:32:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:44:39] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:46:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:49:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 177832 MB (3% inode=67%): [12:50:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:50:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84605 MB (13% inode=99%): [12:52:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121003.000000 [12:53:12] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [12:53:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 143718.000000 [12:53:56] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 18420 [12:53:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:57:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:57:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:31:13] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:31:56] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:31:56] APT on yucca is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:31:57] APT on yarrow is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:32:12] APT on mayapple is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:33:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:38:00] hello all [13:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:46:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:49:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 177402 MB (3% inode=67%): [13:50:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:50:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84393 MB (13% inode=99%): [13:52:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122381.000000 [13:53:14] APT on z-dat-s5-b is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [13:54:14] Heya, DaB [13:54:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 143889.000000 [13:54:55] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20380 [13:54:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:57:39] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:57:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:03:25] @replag [14:03:26] DaBPunkt: s1-rr-a-wd: 45s [+0.00 s/s]; s1-user-wd: 48s [+0.00 s/s]; s2-rr: 5h 44m 2s [+0.45 s/s]; s2-user: 5h 44m 2s [+0.45 s/s]; s2-user-c: error; s2-user-wd: 1d 10h 3m 31s [+0.27 s/s]; s3-user: 22s [-]; s3-user-wd: 44s [+0.00 s/s] [14:03:27] DaBPunkt: s4-user-wd: 1d 15h 48m 20s [-0.35 s/s]; s5-user: 1m 16s [-]; s5-user-c: 2w 3d 5h 14s [+0.35 s/s]; s6-user-wd: 45s [+0.00 s/s]; s7-user-wd: 44s [+0.00 s/s] [14:07:13] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:07:54] APT on z-dat-s2-b is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:07:54] APT on yucca is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:07:54] APT on yarrow is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:08:13] APT on mayapple is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:08:13] APT on z-dat-s5-b is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:33:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:46:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:46:47] [[Special:Log/newusers]] create 10 * Mexicoserver * (New user account) [14:49:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 176994 MB (3% inode=67%): [14:51:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:51:54] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 84068 MB (13% inode=99%): [14:53:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 124038.000000 [14:55:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 140021.000000 [14:55:54] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22679 [14:55:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:57:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:58:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:33:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:46:13] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:49:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 176502 MB (3% inode=67%): [15:51:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83708 MB (13% inode=99%): [15:52:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:53:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 125664.000000 [15:55:56] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 142319.000000 [15:55:56] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24704 [15:55:56] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:57:40] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:58:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:33:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:46:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:49:14] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 175931 MB (3% inode=67%): [16:51:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83417 MB (13% inode=99%): [16:52:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:53:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126383.000000 [16:55:55] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 145127.000000 [16:55:55] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 26213 [16:56:55] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:57:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:58:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:17:11] re [17:34:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:46:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:49:15] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 175587 MB (3% inode=67%): [17:51:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 83122 MB (13% inode=99%): [17:52:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:54:14] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127741.000000 [17:56:54] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 147500.000000 [17:56:55] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28496 [17:57:00] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:57:54] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:58:55] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:21:15] Free Memory on turnera is WARNING: WARNING - 5.3% (444492 kB) free! [18:24:15] Free Memory on turnera is CRITICAL: CRITICAL - 4.9% (409556 kB) free! [18:25:14] Free Memory on turnera is WARNING: WARNING - 5.4% (450264 kB) free! [18:34:54] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:43:15] Free Memory on turnera is CRITICAL: CRITICAL - 4.3% (361476 kB) free! [18:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:46:14] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:49:15] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 175118 MB (3% inode=67%): [18:50:21] PING on amaranth is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 119.00 ms [18:50:32] PING on web.amaranth is WARNING: PING WARNING - Packet loss = 40%, RTA = 118.00 ms [18:51:22] PING on amaranth is WARNING: PING WARNING - Packet loss = 20%, RTA = 119.00 ms [18:52:14] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:52:31] PING on web.amaranth is OK: PING OK - Packet loss = 0%, RTA = 118.00 ms [18:52:55] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82842 MB (13% inode=99%): [18:54:15] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127235.000000 [18:54:26] PING on amaranth is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 119.00 ms [18:57:08] MySQL slave on z-dat-s2-b is CRITICAL: (Return code of 139 is out of bounds) [18:57:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:57:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 150092.000000 [18:58:03] Free Memory on turnera is OK: OK - 90.7% (7601184 kB) free. [18:58:24] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:59:13] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:02:04] Free Memory on turnera is OK: OK - 85.2% (7136012 kB) free. [19:27:14] SMF on web.amaranth is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [19:27:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:28:50] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:40] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2206.000000 [19:30:41] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2172.000000 [19:31:30] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2219.000000 [19:35:29] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:41:50] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [19:42:30] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [19:42:40] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [19:42:50] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [19:43:00] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [19:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:45:00] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2172.000000 [19:45:30] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [19:46:00] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2179.000000 [19:46:00] MySQL slave on cassia is OK: Uptime: 1726984 Threads: 8 Questions: 1595254904 Slow queries: 33191 Opens: 83297 Flush tables: 2 Open tables: 14102 Queries per second avg: 923.723 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [19:46:30] MySQL slave on z-dat-s6-a is OK: Uptime: 2791430 Threads: 8 Questions: 1297855435 Slow queries: 138550 Opens: 4597082 Flush tables: 1 Open tables: 4158 Queries per second avg: 464.942 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1465 [19:46:40] MySQL slave on z-dat-s3-a is OK: Uptime: 928038 Threads: 17 Questions: 1282201761 Slow queries: 56030 Opens: 11652422 Flush tables: 1 Open tables: 16384 Queries per second avg: 1381.626 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1019 [19:46:50] MySQL slave on thyme is OK: Uptime: 1367277 Threads: 16 Questions: 956286542 Slow queries: 96987 Opens: 7732 Flush tables: 1 Open tables: 526 Queries per second avg: 699.409 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1528 [19:46:50] MySQL slave on rosemary is OK: Uptime: 1380568 Threads: 10 Questions: 1192044462 Slow queries: 736456 Opens: 85469 Flush tables: 1 Open tables: 3577 Queries per second avg: 863.444 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1514 [19:47:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:48:00] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2176.000000 [19:48:20] / on ortelius is WARNING: DISK WARNING - free space: / 6207 MB (20% inode=92%): [19:50:00] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 174811 MB (3% inode=67%): [19:51:30] MySQL slave on z-dat-s7-a is OK: Uptime: 2791730 Threads: 6 Questions: 1323345188 Slow queries: 59891 Opens: 4276512 Flush tables: 1 Open tables: 6835 Queries per second avg: 474.23 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 983 [19:53:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:53:40] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3647.000000 [19:53:41] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82490 MB (13% inode=99%): [19:54:40] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3611.000000 [19:55:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130511.000000 [19:55:30] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3659.000000 [20:00:40] SSH on mayapple is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [20:01:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:01:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 153751.000000 [20:01:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27445 [20:01:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:09:00] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3612.000000 [20:10:00] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3619.000000 [20:12:00] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3616.000000 [20:25:00] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3332.000000 [20:25:40] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3422.000000 [20:26:01] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3434.000000 [20:26:01] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3428.000000 [20:27:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:29:41] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1766.000000 [20:30:00] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1708.000000 [20:30:30] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3555.000000 [20:32:00] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1723.000000 [20:32:00] wikidata replag on z-dat-s7-a is OK: QUERY OK: SELECT ts_rc_age() returned 1717.000000 [20:32:40] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3478.000000 [20:34:20] / on ortelius is OK: DISK OK - free space: / 6391 MB (21% inode=92%): [20:35:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:38:30] wikidata replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1743.000000 [20:41:40] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1706.000000 [20:44:41] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:46:21] / on ortelius is WARNING: DISK WARNING - free space: / 6231 MB (20% inode=92%): [20:47:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:50:00] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 173880 MB (3% inode=67%): [20:53:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:53:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 82157 MB (13% inode=99%): [20:55:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 133062.000000 [21:01:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:01:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 156341.000000 [21:01:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 28940 [21:01:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:27:51] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:35:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:44:39] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:46:20] / on ortelius is WARNING: DISK WARNING - free space: / 5752 MB (19% inode=92%): [21:47:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:50:01] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 173961 MB (3% inode=67%): [21:53:01] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [21:53:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81934 MB (13% inode=99%): [21:55:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 135408.000000 [22:01:21] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:01:31] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 158559.000000 [22:01:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31930 [22:01:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:27:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:35:31] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:46:20] / on ortelius is WARNING: DISK WARNING - free space: / 4942 MB (16% inode=92%): [22:47:10] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:50:10] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 173791 MB (3% inode=67%): [22:53:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:53:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81592 MB (13% inode=99%): [22:55:00] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1892.000000 [22:55:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 136877.000000 [22:55:32] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1912 [22:59:30] MySQL slave on z-dat-s5-b is OK: Uptime: 167530 Threads: 4 Questions: 753317094 Slow queries: 1831 Opens: 104274 Flush tables: 1 Open tables: 256 Queries per second avg: 4496.610 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1770 [23:00:00] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1732.000000 [23:01:20] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:01:30] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 161157.000000 [23:01:40] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33352 [23:01:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:50] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:35:30] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:40:30] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2116 [23:41:00] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2137.000000 [23:44:40] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:46:20] / on ortelius is WARNING: DISK WARNING - free space: / 4547 MB (15% inode=92%): [23:47:11] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:51:00] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 170300 MB (3% inode=67%): [23:53:00] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:53:40] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 81231 MB (13% inode=99%): [23:55:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138546.000000