[00:00:04] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40029.000000 [00:01:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37609.000000 [00:01:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39680.000000 [00:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [00:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [00:43:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [00:43:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:43:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126079.000000 [00:43:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [00:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121028.000000 [00:44:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 106263 MB (17% inode=99%): [00:44:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:44:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:46:04] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30186 [00:49:03] / on damiana is WARNING: NRPE: Unable to read output [00:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [00:49:22] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [00:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [00:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:50:04] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:50:23] aliasd on nightshade is CRITICAL: Connection refused [00:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 209621 MB (3% inode=66%): [00:50:23] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:50:44] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [00:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85489.000000 [00:59:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43000.000000 [00:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23709.000000 [01:00:04] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26179.000000 [01:01:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27406.000000 [01:01:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31021.000000 [01:04:34] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [01:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:37:36] @replag [01:37:39] Merlissimo: s1-rr-a: 19s [-]; s1-rr-a-wd: 1d 6h 42m 55s [-3.17 s/s]; s1-user: 12s [-0.01 s/s]; s1-user-wd: 1d 6h 7m 56s [-2.48 s/s]; s2-rr: 9h 11m 9s [+0.37 s/s]; s2-user: 9h 11m 9s [+0.37 s/s]; s2-user-c: error; s2-user-wd: 23h 49m 59s [+0.30 s/s] [01:37:40] Merlissimo: s3-user: 5m 11s [+0.03 s/s]; s3-user-wd: 6h 22m 26s [-1.76 s/s]; s4-user-wd: 11h 33m 56s [-0.03 s/s]; s5-user: 1d 20h 35m 2s [+1.00 s/s]; s5-user-c: error; s6-user-wd: 5h 50m 47s [-1.94 s/s]; s7-user-wd: 7h 46m 12s [-1.29 s/s] [01:43:06] BTW, what do "-a" and "-wd" abbreviate? [01:43:44] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [01:43:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [01:43:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:43:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109084.000000 [01:43:54] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [01:44:04] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 107328.000000 [01:44:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105567 MB (17% inode=99%): [01:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:44:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:46:03] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33524 [01:46:26] -wd is wikidata. [01:46:38] I don't remember what -a is. It may be in channel logs. [01:48:43] Free Memory on damiana is WARNING: WARNING - 6.6% (549284 kB) free! [01:49:03] / on damiana is WARNING: NRPE: Unable to read output [01:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [01:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [01:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:50:04] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:50:14] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:50:24] aliasd on nightshade is CRITICAL: Connection refused [01:50:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 204560 MB (3% inode=66%): [01:50:24] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:50:32] Before searching, I'll wait for when DaBPunkt is around again :-). Thx. [01:50:44] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:50:44] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [01:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 86048.000000 [01:56:12] scfc_de: -wd/-c is wikidata/commons database replag on the database server without this appendix. there can be multiple -rr servers for each cluster. -rr-a is the first one -rr-b would be the second, but there is no cluster having more than one -rr server at the moment [01:59:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 42339.000000 [01:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17996.000000 [02:00:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14111.000000 [02:00:33] Merlissimo: Ah! Thanks. (Though it would be interesting what to do when there were three servers for one cluster - well, not gonna happen :-).) [02:01:04] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17720.000000 [02:01:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25584.000000 [02:01:05] there were two rr server for enwiki in the past for a short time [02:03:14] (i would guess the english alphabet is used, so -rr-c would make sence for a third server) [02:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [02:08:04] Okay, then there would be no collision with the Commons server, and my joke disappears into thin air :-). [02:10:46] scfc_de: if ts admins were greek that won't use -c as next letter ;-) [02:13:43] Free Memory on damiana is CRITICAL: CRITICAL - 4.0% (334992 kB) free! [02:14:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:37:03] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3284.000000 [02:41:03] wikidata replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1428.000000 [02:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [02:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [02:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:43:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 96865.000000 [02:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 94326.000000 [02:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:44] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:44:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:44:54] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [02:45:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105894 MB (17% inode=99%): [02:46:03] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32098 [02:49:03] / on damiana is WARNING: NRPE: Unable to read output [02:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [02:49:22] @replag [02:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:49:23] Merlissimo: s1-rr-a-wd: 1d 2h 25m 7s [-3.59 s/s]; s1-user: 4m 2s [+0.05 s/s]; s1-user-c: 14s [-]; s1-user-wd: 1d 1h 54m 3s [-3.54 s/s]; s2-rr: 8h 58m 0s [-0.18 s/s]; s2-user: 8h 58m 0s [-0.18 s/s]; s2-user-c: error; s2-user-wd: 22h 40m 25s [-0.97 s/s] [02:49:25] Merlissimo: s3-user-wd: 3h 34m 13s [-2.34 s/s]; s4-rr-a: 17s [-]; s4-user: 17s [-]; s4-user-wd: 11h 52m 43s [+0.26 s/s]; s5-user: 1d 21h 46m 48s [+1.00 s/s]; s5-user-c: error; s6-user-wd: 2h 54m 52s [-2.45 s/s]; s7-user-wd: 5h 52m 21s [-1.59 s/s] [02:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [02:50:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:50:04] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:50:23] aliasd on nightshade is CRITICAL: Connection refused [02:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 199166 MB (3% inode=66%): [02:50:23] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:50:43] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [02:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81732.000000 [02:59:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15979.000000 [03:00:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41945.000000 [03:01:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20388.000000 [03:01:03] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8494.000000 [03:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [03:13:43] Free Memory on damiana is CRITICAL: CRITICAL - 2.2% (182504 kB) free! [03:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:20:03] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3445.000000 [03:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [03:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [03:43:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75104.000000 [03:44:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72441.000000 [03:44:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:44:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:44:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [03:45:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105697 MB (17% inode=99%): [03:45:03] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1744.000000 [03:46:04] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35202 [03:47:53] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3556.000000 [03:49:03] / on damiana is WARNING: NRPE: Unable to read output [03:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [03:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:49:44] Load avg. on damiana is WARNING: NRPE: Unable to read output [03:49:53] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3607.000000 [03:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:50:04] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [03:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:50:23] aliasd on nightshade is CRITICAL: Connection refused [03:50:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 198787 MB (3% inode=66%): [03:50:24] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [03:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:50:43] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [03:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84157.000000 [03:55:53] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3551.000000 [04:00:47] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33583.000000 [04:01:04] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15983.000000 [04:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [04:13:43] Free Memory on damiana is CRITICAL: CRITICAL - 3.5% (297316 kB) free! [04:14:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:30:15] [[Wiki server assignments]] ! 10https://wiki.toolserver.org/w/index.php?diff=7835&oldid=7820&rcid=21620 * 91.198.174.202 * (+0) (updated page) [04:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [04:43:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:43:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58443.000000 [04:43:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [04:44:05] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57140.000000 [04:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:44:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:44:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [04:45:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105499 MB (17% inode=99%): [04:46:03] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 35836 [04:49:04] / on damiana is WARNING: NRPE: Unable to read output [04:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [04:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [04:50:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [04:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:23] aliasd on nightshade is CRITICAL: Connection refused [04:50:24] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 198070 MB (3% inode=66%): [04:50:33] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [04:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:50:44] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [04:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82604.000000 [04:55:54] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2292.000000 [04:56:53] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1691.000000 [05:01:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25727.000000 [05:02:03] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9258.000000 [05:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [05:13:43] Free Memory on damiana is CRITICAL: CRITICAL - 1.5% (127284 kB) free! [05:14:23] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:43:44] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [05:43:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:43:44] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50841.000000 [05:43:44] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [05:44:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:44:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:44:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [05:45:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48611.000000 [05:45:04] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105312 MB (17% inode=99%): [05:46:03] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32302 [05:49:03] / on damiana is WARNING: NRPE: Unable to read output [05:49:13] /tmp on damiana is WARNING: NRPE: Unable to read output [05:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:49:43] Load avg. on damiana is WARNING: NRPE: Unable to read output [05:50:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:50:04] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:50:23] aliasd on nightshade is CRITICAL: Connection refused [05:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 200335 MB (3% inode=66%): [05:50:33] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:50:43] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [05:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76161.000000 [05:57:03] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3464.000000 [06:01:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21179.000000 [06:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [06:11:03] wikidata replag on z-dat-s7-a is OK: QUERY OK: SELECT ts_rc_age() returned 1713.000000 [06:13:43] Free Memory on damiana is CRITICAL: CRITICAL - 1.5% (127608 kB) free! [06:27:53] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 3300 MB (16% inode=99%): [06:43:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:43:43] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:43:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46986.000000 [06:43:43] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [06:44:13] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:44:14] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:44:43] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:44:44] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:44:53] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [06:45:03] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40078.000000 [06:45:03] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105175 MB (17% inode=99%): [06:46:03] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30345 [06:49:03] / on damiana is WARNING: NRPE: Unable to read output [06:49:12] /tmp on damiana is WARNING: NRPE: Unable to read output [06:49:13] SMTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:23] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:49:44] Load avg. on damiana is WARNING: NRPE: Unable to read output [06:50:03] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:50:03] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:50:13] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:50:13] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:23] aliasd on nightshade is CRITICAL: Connection refused [06:50:23] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 199984 MB (3% inode=66%): [06:50:33] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:50:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:50:43] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [06:52:03] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73531.000000 [06:54:53] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1990 MB (10% inode=99%): [06:55:53] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2705 MB (13% inode=99%): [07:01:44] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17179.000000 [07:04:33] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [07:13:44] Free Memory on damiana is CRITICAL: CRITICAL - 1.5% (124624 kB) free! [07:22:03] LDAP on ha-ldap.esi is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:26:56] LDAP on ha-ldap.esi is OK: LDAP OK - 2.123 seconds response time [07:27:56] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1971 MB (10% inode=99%): [07:29:46] Free Memory on damiana is OK: OK - 87.3% (7317392 kB) free. [07:29:56] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 2187 MB (11% inode=99%): [07:32:58] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1940 MB (10% inode=99%): [07:35:08] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [07:35:09] / on damiana is WARNING: NRPE: Unable to read output [07:35:16] SSH on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:52] /tmp on damiana is WARNING: NRPE: Unable to read output [07:35:52] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [07:35:52] Load avg. on damiana is WARNING: NRPE: Unable to read output [07:36:09] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:41:09] SSH on nightshade is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [07:41:52] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [07:43:51] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43818.000000 [07:43:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:43:51] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:43:52] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [07:44:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:44:25] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:44:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:44:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:44:58] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [07:45:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33789.000000 [07:45:08] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 105061 MB (17% inode=99%): [07:46:09] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21197 [07:49:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:50:09] SMTP on damiana is CRITICAL: Connection refused [07:50:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:50:09] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:50:16] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:50:16] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:50:51] aliasd on nightshade is CRITICAL: Connection refused [07:50:52] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 199830 MB (3% inode=66%): [07:50:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:50:52] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:50:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [07:52:08] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70742.000000 [08:01:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13276.000000 [08:02:09] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2160.000000 [08:02:51] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2199.000000 [08:02:51] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2198.000000 [08:02:52] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2199.000000 [08:02:58] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2211.000000 [08:13:16] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:26:08] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3601.000000 [08:26:51] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3618.000000 [08:26:51] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3602.000000 [08:26:59] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3651.000000 [08:26:59] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [08:27:09] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2522.000000 [08:27:52] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3600.000000 [08:27:53] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3559.000000 [08:28:08] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1583.000000 [08:31:51] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1505.000000 [08:32:51] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2297 [08:32:51] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2665 [08:33:08] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3166 [08:33:09] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3349 [08:35:51] /tmp on damiana is WARNING: NRPE: Unable to read output [08:35:51] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [08:35:51] MySQL slave on z-dat-s7-a is OK: Uptime: 1541585 Threads: 10 Questions: 550060024 Slow queries: 29454 Opens: 1915816 Flush tables: 1 Open tables: 3776 Queries per second avg: 356.814 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1707 [08:35:51] MySQL slave on z-dat-s6-a is OK: Uptime: 1541584 Threads: 8 Questions: 539374063 Slow queries: 82661 Opens: 2553588 Flush tables: 1 Open tables: 3284 Queries per second avg: 349.883 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1563 [08:35:51] Load avg. on damiana is WARNING: NRPE: Unable to read output [08:36:08] / on damiana is WARNING: NRPE: Unable to read output [08:37:10] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:52] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36846.000000 [08:43:52] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:43:52] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:43:52] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [08:44:10] MySQL slave on thyme is OK: Uptime: 117917 Threads: 14 Questions: 89542518 Slow queries: 11093 Opens: 1061 Flush tables: 1 Open tables: 349 Queries per second avg: 759.369 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1790 [08:44:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:44:51] s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1650.000000 [08:44:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:44:51] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:44:59] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [08:45:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19652.000000 [08:45:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104837 MB (17% inode=99%): [08:46:09] MySQL slave on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [08:49:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:50:08] SMTP on damiana is CRITICAL: Connection refused [08:50:08] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:50:16] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:50:26] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:50:52] aliasd on nightshade is CRITICAL: Connection refused [08:50:52] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 199130 MB (3% inode=66%): [08:50:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:50:52] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:50:52] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:51:09] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [08:51:51] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1755.000000 [08:51:51] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3556.000000 [08:52:13] MySQL slave on rosemary is OK: Uptime: 131683 Threads: 7 Questions: 112297714 Slow queries: 57439 Opens: 9133 Flush tables: 1 Open tables: 2977 Queries per second avg: 852.788 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1701 [08:52:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [08:53:51] aliasd on nightshade is OK: TCP OK - 0.002 second response time on port 984 [500 Not found.] [08:54:52] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1432.000000 [09:12:25] s2 down? [09:12:28] liangent@nightshade:~$ sql zhwiki_p [09:12:29] ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 [09:13:16] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:09] I'm getting the same error. [09:14:16] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:26:59] MySQL on z-dat-s2-b is CRITICAL: Cant connect to MySQL server on z-dat-s2-b (146) [09:26:59] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7224.000000 [09:27:51] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2137.000000 [09:28:10] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2148 [09:30:58] MySQL on z-dat-s2-b is OK: Uptime: 4228 Threads: 11 Questions: 456 Slow queries: 1 Opens: 132 Flush tables: 1 Open tables: 123 Queries per second avg: 0.107 [09:36:09] / on damiana is WARNING: NRPE: Unable to read output [09:36:52] /tmp on damiana is WARNING: NRPE: Unable to read output [09:36:52] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [09:36:52] Load avg. on damiana is WARNING: NRPE: Unable to read output [09:37:10] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:43:51] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23275.000000 [09:43:51] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:43:51] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:43:51] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [09:44:16] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:44:51] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:44:52] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:45:09] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8596.000000 [09:45:10] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104573 MB (17% inode=99%): [09:45:58] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [09:46:10] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27285 [09:50:09] SMTP on damiana is CRITICAL: Connection refused [09:50:09] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:50:16] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:50:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:50:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 198263 MB (3% inode=66%): [09:50:51] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:50:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:50:51] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:51:09] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:51:51] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [09:52:09] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77377.000000 [10:10:10] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3376.000000 [10:13:51] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1775.000000 [10:14:09] MySQL slave on rosemary is OK: Uptime: 136603 Threads: 25 Questions: 120066334 Slow queries: 59579 Opens: 9368 Flush tables: 1 Open tables: 2975 Queries per second avg: 878.943 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1745 [10:14:16] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:14:17] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:17:10] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1618.000000 [10:23:58] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3569.000000 [10:36:10] / on damiana is WARNING: NRPE: Unable to read output [10:36:59] Load avg. on damiana is WARNING: NRPE: Unable to read output [10:37:17] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:51] /tmp on damiana is WARNING: NRPE: Unable to read output [10:37:51] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [10:44:17] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:44:50] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11547.000000 [10:44:50] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:44:50] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:44:50] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [10:45:09] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104325 MB (17% inode=99%): [10:45:50] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:45:50] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:45:59] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [10:45:59] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1683.000000 [10:46:13] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15324 [10:50:11] SMTP on damiana is CRITICAL: Connection refused [10:50:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:50:17] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:50:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:50:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:51:12] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:51:12] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:51:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 197370 MB (3% inode=66%): [10:51:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:52:12] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [10:52:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80978.000000 [11:07:50] wikidata replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3485.000000 [11:14:21] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:14:26] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:16:49] wikidata replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1602.000000 [11:36:10] / on damiana is WARNING: NRPE: Unable to read output [11:37:26] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:56] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [11:37:56] Load avg. on damiana is WARNING: NRPE: Unable to read output [11:38:49] /tmp on damiana is WARNING: NRPE: Unable to read output [11:44:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:44:57] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:44:57] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [11:45:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 104065 MB (17% inode=99%): [11:45:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:45:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:46:10] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [11:47:14] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5357 [11:50:18] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:50:26] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:50:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:50:50] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70155 MB (7% inode=99%): [11:50:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:51:10] SMTP on damiana is CRITICAL: Connection refused [11:51:49] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 195971 MB (3% inode=66%): [11:51:49] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:52:11] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:52:11] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:53:10] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [11:53:10] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84638.000000 [12:06:10] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3423 [12:14:10] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3611 [12:14:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:14:26] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:09] / on damiana is WARNING: NRPE: Unable to read output [12:37:26] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:57] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [12:37:57] Load avg. on damiana is WARNING: NRPE: Unable to read output [12:38:50] /tmp on damiana is WARNING: NRPE: Unable to read output [12:44:26] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:44:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:44:57] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:45:10] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [12:45:56] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:45:56] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:46:11] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [12:46:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103815 MB (17% inode=99%): [12:50:27] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:50:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:51:10] SMTP on damiana is CRITICAL: Connection refused [12:51:10] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:51:50] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 195841 MB (3% inode=66%): [12:51:50] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:51:57] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:52:10] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:52:10] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:53:10] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [12:53:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 88238.000000 [13:14:10] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6501 [13:14:26] SSH on mayapple is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:36:27] SSH on mayapple is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [13:37:10] / on damiana is WARNING: NRPE: Unable to read output [13:37:27] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:37:56] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [13:37:57] Load avg. on damiana is WARNING: NRPE: Unable to read output [13:39:49] /tmp on damiana is WARNING: NRPE: Unable to read output [13:44:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:44:27] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:56] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:45:10] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:45:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:45:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:45:57] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [13:46:10] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [13:46:10] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103573 MB (17% inode=99%): [13:50:36] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:51:10] SMTP on damiana is CRITICAL: Connection refused [13:51:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:51:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:52:18] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:52:18] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:52:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 194548 MB (3% inode=66%): [13:52:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:53:12] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [13:53:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 91838.000000 [13:56:12] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3253 [14:37:12] / on damiana is WARNING: NRPE: Unable to read output [14:37:29] NTP on damiana is CRITICAL: NTP CRITICAL: No response from NTP server [14:37:58] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [14:37:58] Load avg. on damiana is WARNING: NRPE: Unable to read output [14:39:52] /tmp on damiana is WARNING: NRPE: Unable to read output [14:44:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:44:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:45:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:45:59] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:45:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:45:59] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [14:46:12] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [14:46:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103342 MB (17% inode=99%): [14:50:38] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:51:12] SMTP on damiana is CRITICAL: Connection refused [14:51:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:52:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:52:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 203606 MB (3% inode=66%): [14:52:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:53:12] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:53:12] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:54:11] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [14:54:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 95501.000000 [14:57:12] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3593 [14:58:11] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3649 [15:14:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:37:39] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:37:58] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [15:37:59] Load avg. on damiana is WARNING: NRPE: Unable to read output [15:38:11] / on damiana is WARNING: NRPE: Unable to read output [15:39:51] /tmp on damiana is WARNING: NRPE: Unable to read output [15:44:39] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:44:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:45:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:45:59] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:45:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:45:59] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [15:46:12] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [15:46:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 103109 MB (16% inode=99%): [15:50:38] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:51:12] SMTP on damiana is CRITICAL: Connection refused [15:51:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:51:58] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [15:52:52] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 202496 MB (3% inode=66%): [15:52:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:53:11] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:53:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:53:11] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:54:12] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [15:54:13] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 99101.000000 [15:59:11] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6464 [16:37:59] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [16:37:59] Load avg. on damiana is WARNING: NRPE: Unable to read output [16:38:12] / on damiana is WARNING: NRPE: Unable to read output [16:38:38] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:39:52] /tmp on damiana is WARNING: NRPE: Unable to read output [16:44:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:44:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:45:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:45:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:46:59] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:46:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:46:59] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [16:47:12] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [16:47:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102830 MB (16% inode=99%): [16:50:38] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:51:12] SMTP on damiana is CRITICAL: Connection refused [16:51:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:51:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:53:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:53:12] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:53:18] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:53:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 199211 MB (3% inode=66%): [16:53:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:54:12] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [16:54:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102701.000000 [16:59:12] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5572 [17:32:11] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3500 [17:37:59] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [17:37:59] Load avg. on damiana is WARNING: NRPE: Unable to read output [17:38:11] / on damiana is WARNING: NRPE: Unable to read output [17:38:38] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:39:12] MySQL slave on z-dat-s2-b is OK: Uptime: 33519 Threads: 10 Questions: 22076478 Slow queries: 764 Opens: 330991 Flush tables: 1 Open tables: 255 Queries per second avg: 658.625 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1482 [17:39:51] /tmp on damiana is WARNING: NRPE: Unable to read output [17:44:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:45:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:46:00] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:46:58] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:46:58] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [17:46:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:47:12] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [17:47:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102569 MB (16% inode=99%): [17:50:39] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:51:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:51:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:52:11] SMTP on damiana is CRITICAL: Connection refused [17:53:11] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:53:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:53:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 199055 MB (3% inode=66%): [17:53:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:54:11] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [17:54:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 106301.000000 [17:54:18] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:14:12] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:38:11] / on damiana is WARNING: NRPE: Unable to read output [18:38:38] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:38:59] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [18:38:59] Load avg. on damiana is WARNING: NRPE: Unable to read output [18:39:51] /tmp on damiana is WARNING: NRPE: Unable to read output [18:40:11] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2135 [18:45:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:45:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:46:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:46:59] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:46:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:46:59] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [18:48:11] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [18:48:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102335 MB (16% inode=99%): [18:51:38] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:52:11] SMTP on damiana is CRITICAL: Connection refused [18:52:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:52:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:53:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 196519 MB (3% inode=66%): [18:53:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:54:11] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:54:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:55:11] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [18:55:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 109961.000000 [18:55:19] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:08:11] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3653 [19:38:11] / on damiana is WARNING: NRPE: Unable to read output [19:39:30] NTP on damiana is CRITICAL: NTP CRITICAL: No response from NTP server [19:39:51] /tmp on damiana is WARNING: NRPE: Unable to read output [19:39:59] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [19:39:59] Load avg. on damiana is WARNING: NRPE: Unable to read output [19:44:18] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:45:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:45:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:46:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:46:59] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:46:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:46:59] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [19:48:11] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [19:48:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 102072 MB (16% inode=99%): [19:51:39] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:52:12] SMTP on damiana is CRITICAL: Connection refused [19:52:58] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:52:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:53:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 192733 MB (3% inode=66%): [19:53:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:54:11] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:54:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:55:11] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [19:55:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 113561.000000 [19:56:18] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:08:11] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5012 [20:14:51] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2042.000000 [20:34:11] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3328 [20:38:11] / on damiana is WARNING: NRPE: Unable to read output [20:39:39] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:39:52] /tmp on damiana is WARNING: NRPE: Unable to read output [20:39:59] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [20:39:59] Load avg. on damiana is WARNING: NRPE: Unable to read output [20:45:39] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:46:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:46:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:46:59] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:46:59] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [20:46:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:48:13] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [20:48:13] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101852 MB (16% inode=99%): [20:49:12] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3629 [20:51:39] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:52:11] SMTP on damiana is CRITICAL: Connection refused [20:53:52] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 191313 MB (3% inode=66%): [20:53:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:53:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:53:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:54:11] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:54:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:55:10] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [20:55:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 116742.000000 [20:56:19] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:10:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3624.000000 [21:14:10] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:38:12] / on damiana is WARNING: NRPE: Unable to read output [21:39:39] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:40:51] /tmp on damiana is WARNING: NRPE: Unable to read output [21:40:59] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [21:40:59] Load avg. on damiana is WARNING: NRPE: Unable to read output [21:45:39] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:47:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:47:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:47:59] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:47:59] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [21:48:11] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [21:49:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101973 MB (16% inode=99%): [21:50:11] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5435 [21:51:39] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:11] SMTP on damiana is CRITICAL: Connection refused [21:54:10] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:54:11] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:54:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 188557 MB (3% inode=66%): [21:54:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:54:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:54:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:55:11] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:55:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117377.000000 [21:55:51] Free Memory on turnera is WARNING: WARNING - 6.7% (561916 kB) free! [21:57:18] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [21:58:51] Free Memory on turnera is CRITICAL: CRITICAL - 4.5% (375224 kB) free! [22:00:51] Free Memory on turnera is WARNING: WARNING - 5.8% (482452 kB) free! [22:10:52] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4660.000000 [22:14:11] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:15:57] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (384100 kB) free! [22:20:52] Free Memory on turnera is OK: OK - 7.4% (618360 kB) free. [22:28:51] Free Memory on turnera is WARNING: WARNING - 6.3% (524540 kB) free! [22:38:52] Free Memory on turnera is OK: OK - 7.3% (612060 kB) free. [22:39:11] / on damiana is WARNING: NRPE: Unable to read output [22:39:39] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:51] /tmp on damiana is WARNING: NRPE: Unable to read output [22:40:59] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [22:41:58] Load avg. on damiana is WARNING: NRPE: Unable to read output [22:45:39] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:47:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:47:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:47:59] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:47:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:47:59] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [22:48:12] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [22:49:11] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101403 MB (16% inode=99%): [22:50:52] Free Memory on turnera is CRITICAL: CRITICAL - 4.6% (388812 kB) free! [22:51:11] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8013 [22:51:40] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:52:11] SMTP on damiana is CRITICAL: Connection refused [22:54:11] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:54:21] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:54:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 186433 MB (3% inode=66%): [22:54:51] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:54:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:54:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:55:11] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [22:55:12] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 118518.000000 [22:57:18] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:11:51] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5963.000000 [23:14:19] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:39:11] / on damiana is WARNING: NRPE: Unable to read output [23:39:39] NTP on damiana is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:40:51] /tmp on damiana is WARNING: NRPE: Unable to read output [23:40:59] Environment IPMI on damiana is WARNING: NRPE: Unable to read output [23:41:59] Load avg. on damiana is WARNING: NRPE: Unable to read output [23:45:38] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:47:59] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:47:59] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:47:59] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:47:59] APT on yucca is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [23:48:11] APT on z-dat-s2-b is CRITICAL: APT CRITICAL: 3 packages available for upgrade (3 critical updates). [23:49:12] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [23:49:12] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 101152 MB (16% inode=99%): [23:50:51] Free Memory on turnera is CRITICAL: CRITICAL - 2.1% (174912 kB) free! [23:50:59] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 69116 MB (7% inode=99%): [23:51:12] MySQL slave on z-dat-s2-b is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9846 [23:52:11] SMTP on damiana is CRITICAL: Connection refused [23:52:40] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:54:11] APT on yarrow is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:54:19] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:54:51] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 180833 MB (3% inode=67%): [23:54:52] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:54:59] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:54:59] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:55:11] APT on nightshade is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates). [23:56:11] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 120673.000000 [23:57:18] APT on mayapple is CRITICAL: APT CRITICAL: 4 packages available for upgrade (4 critical updates).