[00:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 474126 MB (8% inode=64%): [00:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:01:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68101.000000 [00:01:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 122586 [00:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62818 MB (10% inode=99%): [00:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123642.000000 [00:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:02:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24657.000000 [00:02:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [00:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:06:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8775.000000 [00:13:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122096.000000 [00:31:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:32:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473926 MB (8% inode=64%): [01:01:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:01:12] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:01:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:01:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68065.000000 [01:01:32] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121889 [01:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62591 MB (10% inode=99%): [01:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123593.000000 [01:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:02:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18782.000000 [01:02:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [01:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:04:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:06:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6266.000000 [01:09:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 30324 MB (3% inode=99%): [01:13:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121343.000000 [01:13:32] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1942.000000 [01:13:44] @replag [01:16:56] Betacommand: s1-rr-a-wd: 19h 5m 41s [+0.21 s/s]; s1-user: 1d 9h 44m 40s [+0.34 s/s]; s1-user-c: 4h 31m 0s [+0.01 s/s]; s1-user-wd: 1d 10h 15m 25s [+0.64 s/s]; s2-rr: 20m 40s [+0.03 s/s]; s2-user: 20m 40s [+0.03 s/s]; s2-user-c: error; s2-user-wd: 19h 5m 40s [+0.21 s/s] [01:16:57] Betacommand: s3-user: 4m 55s [+0.01 s/s]; s3-user-wd: 29m 22s [-0.09 s/s]; s4-user-wd: 1h 36m 53s [+0.12 s/s]; s5-rr-a-wd: 1m 30s [-0.02 s/s]; s6-user: 17s [-0.00 s/s]; s6-user-wd: 34m 24s [+0.01 s/s]; s7-user-wd: 4m 49s [-0.02 s/s] [01:31:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:00:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473748 MB (8% inode=64%): [02:01:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:01:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69538.000000 [02:01:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121333 [02:01:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60500 MB (9% inode=99%): [02:01:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122908.000000 [02:01:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:02:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9143.000000 [02:02:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:04:32] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:04:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:06:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5600.000000 [02:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121147.000000 [02:13:33] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3291.000000 [02:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:39:33] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3610.000000 [03:00:52] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:00:52] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473605 MB (8% inode=64%): [03:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:14] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:01:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:01:34] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67845.000000 [03:01:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 62177 MB (10% inode=99%): [03:01:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120638 [03:01:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123350.000000 [03:01:34] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:02:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7276.000000 [03:02:32] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:03:53] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1945 [03:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:04:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:06:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5427.000000 [03:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121169.000000 [03:31:42] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:39:33] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4227.000000 [03:39:53] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 22.90, 29.68, 16.85 [03:41:52] Load avg. on ortelius is WARNING: WARNING - load average: 11.39, 23.14, 15.97 [03:47:53] Load avg. on ortelius is OK: OK - load average: 8.28, 14.30, 14.35 [04:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 475462 MB (8% inode=64%): [04:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9092.000000 [04:02:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68040.000000 [04:02:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 61457 MB (10% inode=99%): [04:02:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 121240 [04:02:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 124224.000000 [04:02:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:02:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:03:53] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3314 [04:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:04:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:06:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4852.000000 [04:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 121436.000000 [04:15:33] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3593.000000 [04:17:03] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1914.000000 [04:17:32] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1928 [04:21:53] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3612 [04:31:33] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2092.000000 [04:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:33:33] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3609.000000 [04:37:39] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3548.000000 [04:56:13] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3565.000000 [05:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 475349 MB (8% inode=64%): [05:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:02:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11901.000000 [05:02:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126383.000000 [05:02:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67893.000000 [05:02:34] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60985 MB (9% inode=99%): [05:02:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123689 [05:02:34] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [05:03:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:04:33] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:13:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123945.000000 [05:17:02] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2488.000000 [05:17:33] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2482 [05:21:53] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5023 [05:30:14] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 128192 MB (20% inode=99%): [05:30:23] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1945.000000 [05:31:33] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2379.000000 [05:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:32:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:37:32] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2635.000000 [05:52:23] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1499.000000 [05:56:13] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2449.000000 [06:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 475151 MB (8% inode=64%): [06:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:02:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14046.000000 [06:02:32] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127252.000000 [06:02:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66335.000000 [06:03:33] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 125359 [06:03:33] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60766 MB (9% inode=99%): [06:03:33] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [06:03:33] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:04:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:05:33] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1750.000000 [06:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 125307.000000 [06:16:33] MySQL slave on daphne is OK: Uptime: 1583383 Threads: 13 Questions: 804425244 Slow queries: 133001 Opens: 68378 Flush tables: 1 Open tables: 1493 Queries per second avg: 508.42 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1671 [06:17:03] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1484.000000 [06:21:53] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6285 [06:27:13] wikidata replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1795.000000 [06:31:13] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 123687 MB (20% inode=99%): [06:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:37:33] wikidata replag on z-dat-s6-a is OK: QUERY OK: SELECT ts_rc_age() returned 1782.000000 [07:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 474739 MB (8% inode=64%): [07:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:02:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16243.000000 [07:02:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64435.000000 [07:02:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 127871.000000 [07:03:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 126553 [07:03:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60511 MB (9% inode=99%): [07:03:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [07:03:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:04:33] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:04:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 126579.000000 [07:21:53] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7890 [07:31:12] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 120100 MB (19% inode=99%): [07:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:32:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:34:53] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3363 [07:36:53] MySQL slave on z-dat-s4-a is OK: Uptime: 6596673 Threads: 1 Questions: 596971346 Slow queries: 346 Opens: 689 Flush tables: 11 Open tables: 229 Queries per second avg: 90.495 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1089 [08:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 474382 MB (8% inode=64%): [08:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:02:22] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18624.000000 [08:02:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63891.000000 [08:02:34] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 128613.000000 [08:03:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60316 MB (9% inode=99%): [08:03:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 125493 [08:03:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:03:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:04:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:04:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:13:13] wikidata replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2054.000000 [08:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 125324.000000 [08:25:08] Marlen Caemmerer * Re: [Toolserver-announce] IP Renumbering of the complete Toolserver cluster [08:31:13] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 116782 MB (19% inode=99%): [08:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:32:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:48:25] @replac [08:53:12] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3620.000000 [08:57:03] Free Memory on damiana is WARNING: WARNING - 6.1% (514820 kB) free! [09:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473894 MB (8% inode=64%): [09:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:02:23] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20937.000000 [09:02:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62262.000000 [09:02:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 129272.000000 [09:03:42] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 60126 MB (9% inode=99%): [09:03:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 124516 [09:03:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [09:03:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:04:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:04:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:05:02] Free Memory on damiana is CRITICAL: CRITICAL - 4.8% (404988 kB) free! [09:13:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 124337.000000 [09:26:03] Free Memory on damiana is OK: OK - 9.3% (780920 kB) free. [09:31:13] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 112162 MB (18% inode=99%): [09:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:32:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:46:33] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 44332 MB (8% inode=99%): [09:52:03] Free Memory on damiana is WARNING: WARNING - 6.3% (527548 kB) free! [09:53:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4778.000000 [10:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473736 MB (8% inode=64%): [10:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:01:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:02:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 60403.000000 [10:02:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 129837.000000 [10:03:33] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23598.000000 [10:03:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59973 MB (9% inode=99%): [10:03:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 123176 [10:03:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:03:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:04:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:04:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:05:03] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (422928 kB) free! [10:13:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 123108.000000 [10:31:13] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 109511 MB (17% inode=99%): [10:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:32:22] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:46:22] @replag [10:49:34] Betacommand: s1-rr-a-wd: 16h 47m 40s [-0.24 s/s]; s1-user: 1d 10h 5m 52s [+0.04 s/s]; s1-user-c: 7h 7m 34s [+0.27 s/s]; s1-user-wd: 1d 12h 16m 34s [+0.21 s/s]; s2-rr: 36s [-0.04 s/s]; s2-user: 36s [-0.04 s/s]; s2-user-c: error; s2-user-wd: 16h 47m 40s [-0.24 s/s] [10:49:35] Betacommand: s3-user-wd: 19m 34s [-0.02 s/s]; s4-user-wd: 1h 39m 50s [+0.01 s/s]; s5-rr-a-wd: 2m 14s [+0.00 s/s]; s5-user: 7m 28s [+0.00 s/s]; s5-user-c: 6m 43s [+0.00 s/s]; s6-user-wd: 2m 14s [-0.06 s/s]; s7-user-wd: 2m 14s [-0.00 s/s] [10:53:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6017.000000 [11:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 473367 MB (8% inode=64%): [11:01:12] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:01:22] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61118.000000 [11:03:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130631.000000 [11:03:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26257.000000 [11:03:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59730 MB (9% inode=99%): [11:03:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 122692 [11:03:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:03:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:04:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:04:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:05:03] Free Memory on damiana is CRITICAL: CRITICAL - 1.9% (156796 kB) free! [11:13:22] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 122373.000000 [11:31:13] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 107066 MB (17% inode=99%): [11:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:53:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7189.000000 [12:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 472992 MB (8% inode=64%): [12:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:33] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63172.000000 [12:03:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 130970.000000 [12:03:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28430.000000 [12:03:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 120062 [12:03:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59470 MB (9% inode=99%): [12:03:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:03:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:04:42] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:04:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:05:03] Free Memory on damiana is CRITICAL: CRITICAL - 1.6% (130412 kB) free! [12:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 119635.000000 [12:31:13] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 103878 MB (16% inode=99%): [12:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:53:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9292.000000 [13:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:00:54] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 472754 MB (8% inode=64%): [13:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:02:32] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64433.000000 [13:03:33] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 131637.000000 [13:03:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31128.000000 [13:03:43] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 117299 [13:03:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59235 MB (9% inode=99%): [13:03:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:03:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:04:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:05:03] Free Memory on damiana is CRITICAL: CRITICAL - 1.5% (129648 kB) free! [13:05:42] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:09:03] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 29364 MB (3% inode=99%): [13:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 117467.000000 [13:31:03] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2014 [13:31:13] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 101386 MB (16% inode=99%): [13:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:37:03] MySQL slave on z-dat-s2-b is OK: Uptime: 310437 Threads: 10 Questions: 491236095 Slow queries: 5273 Opens: 2991049 Flush tables: 1 Open tables: 256 Queries per second avg: 1582.401 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1760 [13:53:13] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11509.000000 [14:00:53] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:00:53] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 472520 MB (8% inode=64%): [14:01:13] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:13] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:01:23] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:43] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66081.000000 [14:03:32] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 132318.000000 [14:03:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33788.000000 [14:03:42] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 113857 [14:03:43] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 59009 MB (9% inode=99%): [14:03:43] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [14:03:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:04:43] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:02] Free Memory on damiana is CRITICAL: CRITICAL - 1.5% (128568 kB) free! [14:05:43] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:13:23] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 114023.000000 [14:30:03] Sun Grid Engine execd on mayapple is UNKNOWN: Execution timeout exceeded [14:30:13] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [14:30:23] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [14:31:13] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 98480 MB (16% inode=99%): [14:31:43] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:31:52] HTTP proxy on ha-proxy.esi is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:12] Authoritative DNS on ha-dns-auth is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:32:23] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:37:24] wikidata replag on z-dat-s7-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2237.000000 [14:37:24] wikidata replag on z-dat-s6-a is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2237.000000 [14:37:24] wikidata replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2237.000000 [14:37:54] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [14:41:06] HTTP proxy on ha-proxy.esi is OK: HTTP OK: HTTP/1.0 302 Moved Temporarily - 1114 bytes in 0.181 second response time [14:41:16] Sun Grid Engine execd on ortelius is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:41:16] Authoritative DNS on ha-dns-auth is OK: DNS OK: 0.055 seconds response time. 1.www.toolserver.org returns 91.198.174.203 [14:41:17] Sun Grid Engine execd on willow is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:41:33] MySQL on ha-sql.esi is CRITICAL: Access denied for user tsnagios7643@turnera-bge0 (using password: YES) [14:41:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:41:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:41:52] Sun Grid Engine execd on mayapple is CRITICAL: CRITICAL: execd not communicating [14:42:03] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [14:42:12] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [14:42:53] Sun Grid Engine execd on mayapple is OK: Host and Queues Ok [14:43:23] Free Memory on damiana is OK: OK - 92.4% (7740696 kB) free. [14:45:30] toolserver.org seems dead, I got only timeout since 30 mins [14:46:33] MySQL on ha-sql.esi is OK: Uptime: 39 Threads: 1 Questions: 137 Slow queries: 1 Opens: 19 Flush tables: 1 Open tables: 12 Queries per second avg: 3.512 [14:47:19] it's back from time to time [14:53:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14377.000000 [15:00:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3640.000000 [15:00:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3641.000000 [15:00:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3641.000000 [15:01:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:01:04] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 472182 MB (8% inode=64%): [15:01:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:01:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:01:32] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:47] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2214.000000 [15:02:47] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2217.000000 [15:02:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67931.000000 [15:03:47] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2213.000000 [15:03:47] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2213.000000 [15:03:47] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36412.000000 [15:03:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58789 MB (9% inode=99%): [15:03:48] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 134635.000000 [15:03:48] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 112181 [15:03:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:04:53] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:06:53] SSH on nightshade is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:07:23] Sun Grid Engine execd on nightshade is UNKNOWN: Execution timeout exceeded [15:08:14] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2177.000000 [15:11:11] $ ssh willow [15:11:11] get_socket_address: getnameinfo 8 failed: Name or service not known [15:11:16] related to IP change? [15:13:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110921.000000 [15:26:46] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [15:26:46] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3657.000000 [15:27:14] Sun Grid Engine execd on nightshade is OK: Host and Queues Ok [15:27:46] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3653.000000 [15:27:46] wikidata replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3653.000000 [15:27:46] SSH on nightshade is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [15:31:14] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 96782 MB (15% inode=99%): [15:31:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:32:14] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3618.000000 [15:32:15] SSH on yarrow is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:32:33] Sun Grid Engine execd on yarrow is UNKNOWN: Execution timeout exceeded [15:38:04] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3394 [15:39:08] Johannes Kroll * [Toolserver-l] TS web outage? [15:41:14] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3590.000000 [15:41:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:41:46] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2924.000000 [15:41:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:42:03] MySQL slave on z-dat-s4-a is OK: Uptime: 6625784 Threads: 1 Questions: 598729529 Slow queries: 346 Opens: 689 Flush tables: 11 Open tables: 229 Queries per second avg: 90.363 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1367 [15:43:45] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1335.000000 [15:43:45] wikidata replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3468.000000 [15:45:45] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1882 [15:46:15] MySQL slave on z-dat-s3-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3471 [15:46:15] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1625.000000 [15:46:15] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3540 [15:46:15] MySQL slave on z-dat-s6-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2746 [15:46:46] MySQL slave on daphne is OK: Uptime: 1617595 Threads: 4 Questions: 818996443 Slow queries: 134818 Opens: 69611 Flush tables: 1 Open tables: 1520 Queries per second avg: 506.305 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1309 [15:47:04] MySQL slave on z-dat-s5-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2321 [15:47:14] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3646 [15:48:14] MySQL slave on z-dat-s7-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3472 [15:49:05] MySQL slave on z-dat-s5-b is OK: Uptime: 320761 Threads: 3 Questions: 540693577 Slow queries: 2989 Opens: 236253 Flush tables: 1 Open tables: 255 Queries per second avg: 1685.658 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1764 [15:49:08] Johannes Kroll * Re: [Toolserver-l] TS web outage? [15:49:15] MySQL slave on z-dat-s6-a is OK: Uptime: 4505190 Threads: 7 Questions: 2585978632 Slow queries: 267070 Opens: 7875694 Flush tables: 1 Open tables: 4276 Queries per second avg: 573.999 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1683 [15:49:47] wikidata replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1568.000000 [15:53:32] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17976.000000 [15:54:13] MySQL slave on z-dat-s2-b is OK: Uptime: 318668 Threads: 14 Questions: 498906804 Slow queries: 5360 Opens: 3047327 Flush tables: 1 Open tables: 256 Queries per second avg: 1565.600 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1604 [15:55:13] MySQL slave on z-dat-s3-a is OK: Uptime: 2642152 Threads: 22 Questions: 3854722251 Slow queries: 198921 Opens: 26341341 Flush tables: 1 Open tables: 16384 Queries per second avg: 1458.932 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1756 [15:58:14] MySQL slave on z-dat-s7-a is OK: Uptime: 4505731 Threads: 4 Questions: 2493260397 Slow queries: 106622 Opens: 6551009 Flush tables: 1 Open tables: 7041 Queries per second avg: 553.353 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1657 [16:01:03] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:01:03] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 471909 MB (8% inode=64%): [16:01:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:01:24] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:01:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7300.000000 [16:01:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7301.000000 [16:01:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7301.000000 [16:02:23] Sun Grid Engine execd on yarrow is OK: Host and Queues Ok [16:03:06] SSH on yarrow is OK: SSH OK - OpenSSH_5.5p1 Debian-6+squeeze3 (protocol 2.0) [16:03:47] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40013.000000 [16:03:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 71591.000000 [16:03:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 111616 [16:03:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 138236.000000 [16:03:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58544 MB (9% inode=99%): [16:04:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [16:04:54] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:05:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:14:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 110738.000000 [16:26:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7255.000000 [16:26:47] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7258.000000 [16:31:15] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 96791 MB (15% inode=99%): [16:31:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:42:44] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:42:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:53:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21577.000000 [17:01:06] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:01:06] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 471511 MB (8% inode=64%): [17:01:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:01:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:02:48] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10960.000000 [17:02:48] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10960.000000 [17:02:48] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10961.000000 [17:04:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 108256 [17:04:47] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43413.000000 [17:04:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75252.000000 [17:04:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 141896.000000 [17:04:48] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58321 MB (9% inode=99%): [17:04:48] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:05:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:14:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 107904.000000 [17:26:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10855.000000 [17:26:47] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10858.000000 [17:31:14] /sql/data/commonswiki on rosemary is WARNING: DISK WARNING - free space: /sql/data/commonswiki 95021 MB (15% inode=99%): [17:31:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:42:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:42:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:53:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25177.000000 [18:01:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:01:05] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 471040 MB (8% inode=64%): [18:01:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:01:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:01:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:02:14] /sql/data/commonswiki on rosemary is OK: DISK OK - free space: /sql/data/commonswiki 156969 MB (25% inode=99%): [18:03:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14621.000000 [18:03:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14621.000000 [18:03:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14621.000000 [18:04:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44912.000000 [18:05:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:05:45] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78911.000000 [18:05:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 58059 MB (9% inode=99%): [18:05:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 106428 [18:05:46] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 145556.000000 [18:05:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [18:06:54] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:14:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 105748.000000 [18:26:47] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14455.000000 [18:26:47] s4 replag on z-dat-s5-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14458.000000 [18:31:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:42:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:42:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:53:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27249.000000 [19:01:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:01:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:02:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:02:08] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 470558 MB (8% inode=64%): [19:02:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:46] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15981.000000 [19:04:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15980.000000 [19:04:46] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15980.000000 [19:04:53] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30982.000000 [19:05:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79231.000000 [19:05:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 105027 [19:05:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 147583.000000 [19:05:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57812 MB (9% inode=99%): [19:05:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [19:06:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:14:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 104435.000000 [19:21:44] s4 replag on z-dat-s5-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3471.000000 [19:27:45] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4106.000000 [19:28:45] s4 replag on z-dat-s5-b is OK: QUERY OK: SELECT ts_rc_age() returned 1684.000000 [19:30:46] s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3428.000000 [19:31:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:36:47] s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1689.000000 [19:43:45] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:43:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:53:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26694.000000 [20:01:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:01:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:02:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:03:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:03:06] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 470279 MB (8% inode=64%): [20:04:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15392.000000 [20:04:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15368.000000 [20:04:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15367.000000 [20:05:04] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:05:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76757.000000 [20:05:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57582 MB (9% inode=99%): [20:05:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 147010.000000 [20:05:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 102914 [20:05:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19473.000000 [20:06:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:07:58] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:14:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103149.000000 [20:31:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:43:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:43:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:53:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26277.000000 [21:01:25] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:01:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:02:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:04:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:04:05] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 469729 MB (8% inode=64%): [21:04:46] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14405.000000 [21:04:48] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14424.000000 [21:04:48] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14406.000000 [21:05:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:47] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75503.000000 [21:06:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 102924 [21:06:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 146254.000000 [21:06:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57320 MB (9% inode=99%): [21:06:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:06:54] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11557.000000 [21:07:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:14:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 103151.000000 [21:31:54] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:43:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:43:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:46:46] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 43826 MB (8% inode=99%): [21:53:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25705.000000 [22:00:53] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3477.000000 [22:01:33] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:02:23] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:02:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:04:05] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:04:05] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 468838 MB (8% inode=64%): [22:05:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13316.000000 [22:05:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13343.000000 [22:05:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14799.000000 [22:06:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:07:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76934.000000 [22:07:46] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 102234 [22:07:46] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 145257.000000 [22:07:46] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 57101 MB (9% inode=99%): [22:07:46] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:08:53] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:11:46] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1429 MB (11% inode=99%): [22:12:45] /tmp on ortelius is CRITICAL: DISK CRITICAL - free space: /tmp 1137 MB (9% inode=99%): [22:12:54] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1644.000000 [22:14:47] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102060.000000 [22:16:46] /tmp on ortelius is WARNING: DISK WARNING - free space: /tmp 1513 MB (11% inode=99%): [22:43:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:43:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:53:33] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 24970.000000 [23:01:34] NTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:01:47] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:02:24] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:02:33] SMTP on yucca is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:04] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:05:05] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 468387 MB (8% inode=64%): [23:06:05] NTP on sage is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:06:47] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12570.000000 [23:06:47] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12570.000000 [23:06:47] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16068.000000 [23:08:46] wikidata replag on z-dat-s2-b is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 78596.000000 [23:08:47] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 102456 [23:08:47] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 143590.000000 [23:08:47] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 56843 MB (9% inode=99%): [23:08:47] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:08:57] RAID on daphne is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:15:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 102464.000000 [23:43:47] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:43:47] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:54:34] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23916.000000