[00:09:32] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6920.000000 [00:09:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7488.000000 [00:09:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6921.000000 [00:10:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7108.000000 [00:34:12] Free Memory on damiana is WARNING: WARNING - 7.0% (587560 kB) free! [00:35:12] Free Memory on damiana is OK: OK - 7.3% (613140 kB) free. [00:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 153928 MB (2% inode=52%): [00:36:12] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:36:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:36:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23384.000000 [00:36:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:36:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240806.000000 [00:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97734 MB (16% inode=99%): [00:36:43] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.95, 22.64, 22.93 [00:36:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8926.000000 [00:36:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [00:36:53] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23380 [00:36:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:46:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8859.000000 [01:09:32] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8090.000000 [01:09:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8801.000000 [01:09:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8104.000000 [01:10:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8246.000000 [01:32:12] Free Memory on damiana is CRITICAL: CRITICAL - 5.0% (419460 kB) free! [01:33:12] Free Memory on damiana is WARNING: WARNING - 5.5% (461048 kB) free! [01:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 152974 MB (2% inode=52%): [01:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:36:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:36:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:36:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22903.000000 [01:36:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:36:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240642.000000 [01:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97312 MB (16% inode=99%): [01:36:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.04, 24.50, 24.04 [01:36:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9943.000000 [01:36:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:36:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22911 [01:36:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:38:12] Free Memory on damiana is OK: OK - 7.2% (601016 kB) free. [01:46:41] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10306.000000 [02:09:32] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9981.000000 [02:09:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10984.000000 [02:09:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9992.000000 [02:10:22] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10266.000000 [02:10:27] is there any machine which has htpasswd installed? [02:19:24] nighty [02:23:02] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 73512 MB (7% inode=99%): [02:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 152785 MB (2% inode=52%): [02:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:36:13] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:36:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:36:33] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19548.000000 [02:36:33] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:36:33] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1240436.000000 [02:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97544 MB (16% inode=99%): [02:36:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.94, 23.98, 24.80 [02:36:52] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12033.000000 [02:36:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:36:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [02:36:52] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19542 [02:36:52] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:36:52] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [02:46:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12297.000000 [03:09:32] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11603.000000 [03:09:42] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 12920.000000 [03:09:42] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11612.000000 [03:10:21] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11929.000000 [03:26:41] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [03:26:42] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [03:26:51] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [03:27:22] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [03:27:22] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [03:27:32] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [03:27:32] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [03:36:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 149168 MB (2% inode=52%): [03:36:12] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:36:32] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:36:32] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18633.000000 [03:36:32] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:36:32] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1241829.000000 [03:36:42] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97400 MB (16% inode=99%): [03:36:42] Load avg. on willow is CRITICAL: CRITICAL - load average: 25.76, 25.95, 26.33 [03:36:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:36:53] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14473.000000 [03:36:53] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:36:53] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [03:36:53] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [03:46:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15031.000000 [03:51:32] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2165.000000 [03:51:42] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2180.000000 [03:52:12] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2209.000000 [03:52:22] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2211.000000 [04:09:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 16308.000000 [04:09:46] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15031.000000 [04:10:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15370.000000 [04:10:26] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15073.000000 [04:15:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3605.000000 [04:15:45] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3620.000000 [04:16:25] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3655.000000 [04:16:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3654.000000 [04:26:46] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [04:26:46] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [04:27:03] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [04:27:25] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [04:27:25] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [04:27:35] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [04:27:35] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [04:36:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:36:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:36:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1245429.000000 [04:36:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97330 MB (16% inode=99%): [04:36:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 27.77, 26.32, 26.62 [04:36:55] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [04:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:36:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [04:36:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 149528 MB (2% inode=52%): [04:37:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:37:26] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19051.000000 [04:37:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18132.000000 [04:47:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18691.000000 [05:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:09:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19909.000000 [05:09:46] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18631.000000 [05:10:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18970.000000 [05:10:25] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18673.000000 [05:15:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7206.000000 [05:15:45] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7221.000000 [05:16:25] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7255.000000 [05:16:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7255.000000 [05:26:45] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [05:26:45] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [05:26:55] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [05:27:26] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [05:27:26] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [05:27:35] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [05:27:35] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [05:36:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:36:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:36:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1249029.000000 [05:36:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97233 MB (15% inode=99%): [05:36:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.99, 26.71, 27.14 [05:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:36:55] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [05:36:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:36:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 149319 MB (2% inode=52%): [05:37:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19255.000000 [05:37:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:37:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21732.000000 [05:47:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22292.000000 [06:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [06:09:44] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23509.000000 [06:09:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22231.000000 [06:10:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22570.000000 [06:10:25] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22274.000000 [06:15:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10806.000000 [06:15:45] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10823.000000 [06:16:25] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10855.000000 [06:16:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10855.000000 [06:26:45] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [06:26:45] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [06:26:55] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [06:27:26] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [06:27:26] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [06:27:36] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [06:27:36] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [06:36:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:36:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:36:36] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1252630.000000 [06:36:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97148 MB (15% inode=99%): [06:36:45] Load avg. on willow is CRITICAL: CRITICAL - load average: 26.42, 26.54, 26.67 [06:36:56] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:36:56] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [06:36:56] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:36:56] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 149117 MB (2% inode=52%): [06:37:28] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:37:28] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17702.000000 [06:37:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25332.000000 [06:47:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25891.000000 [07:04:55] wikidata replag is rather high... [07:06:44] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:09:53] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27109.000000 [07:09:53] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25830.000000 [07:10:26] Free Memory on damiana is WARNING: WARNING - 6.2% (516912 kB) free! [07:10:26] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26171.000000 [07:10:26] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25873.000000 [07:15:36] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14409.000000 [07:15:45] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14423.000000 [07:16:25] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14455.000000 [07:16:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14455.000000 [07:17:25] Free Memory on damiana is OK: OK - 7.3% (614464 kB) free. [07:26:45] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [07:26:45] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [07:26:55] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [07:27:25] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [07:27:25] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [07:27:35] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [07:27:35] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [07:36:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:36:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1256230.000000 [07:36:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:36:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 97001 MB (15% inode=99%): [07:36:46] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.20, 23.57, 23.91 [07:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:37:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28932.000000 [07:37:55] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [07:37:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:37:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 148817 MB (2% inode=52%): [07:38:25] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:38:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 15782.000000 [07:47:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29492.000000 [07:49:25] Free Memory on damiana is WARNING: WARNING - 6.2% (522028 kB) free! [07:55:26] Free Memory on damiana is OK: OK - 7.9% (663380 kB) free. [08:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:09:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30708.000000 [08:09:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29431.000000 [08:10:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29770.000000 [08:10:25] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29473.000000 [08:15:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18008.000000 [08:15:45] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18023.000000 [08:16:25] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18055.000000 [08:16:25] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18055.000000 [08:19:39] @replag [08:19:40] duh: s1-rr-a: 5h 4m 19s [+0.02 s/s]; s1-rr-a-wd: 8h 20m 30s [+0.46 s/s]; s1-user: 5h 4m 19s [-0.45 s/s]; s1-user-c: 5h 4m 20s [+0.03 s/s]; s1-user-wd: 8h 43m 39s [+0.48 s/s]; s2-user: 5h 4m 20s [+0.24 s/s]; s2-user-c: 2w 13h 40m 21s [+0.37 s/s]; s2-user-wd: 8h 44m 11s [+0.48 s/s] [08:19:41] duh: s3-user: 5h 4m 46s [+0.28 s/s]; s4-rr-a: 5h 4m 20s [+0.01 s/s]; s4-user: 5h 4m 20s [+0.01 s/s]; s4-user-wd: 8h 25m 34s [+0.47 s/s]; s5-rr-a: 5h 4m 25s [+0.01 s/s]; s5-user: 5h 4m 25s [+0.01 s/s]; s5-user-c: 2w 13h 40m 21s [+0.37 s/s]; s5-user-wd: 8h 44m 11s [+0.48 s/s] [08:19:42] duh: s6-user: 5h 4m 57s [+0.02 s/s]; s6-user-wd: 8h 41m 49s [+0.48 s/s]; s7-user: 5h 4m 22s [+0.01 s/s]; s7-user-wd: 8h 20m 30s [+0.46 s/s] [08:19:45] hmmmm [08:19:58] its slowly growing [08:23:28] https://toolserver.org/~bryan/stats/replag/ [08:26:45] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [08:26:45] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [08:26:55] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [08:27:26] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [08:27:26] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [08:27:35] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [08:27:36] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [08:36:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:36:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1259829.000000 [08:36:36] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:36:46] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96830 MB (15% inode=99%): [08:36:46] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.42, 23.55, 23.59 [08:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:37:45] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32533.000000 [08:37:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [08:37:55] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [08:37:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 148777 MB (2% inode=52%): [08:38:25] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 19383.000000 [08:38:26] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:47:35] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33091.000000 [09:06:45] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:09:45] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34309.000000 [09:09:45] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33030.000000 [09:10:25] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33370.000000 [09:10:35] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33075.000000 [09:15:35] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21609.000000 [09:15:45] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21623.000000 [09:16:25] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21655.000000 [09:16:26] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21654.000000 [09:26:45] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [09:26:46] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [09:26:54] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [09:27:36] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [09:27:36] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [09:28:25] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [09:28:26] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [09:36:25] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:36:35] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1263429.000000 [09:36:35] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:36:45] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96325 MB (15% inode=99%): [09:36:46] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.50, 23.01, 23.52 [09:36:55] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:37:44] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36132.000000 [09:37:54] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:37:55] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:37:55] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 148270 MB (2% inode=52%): [09:38:35] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 22984.000000 [09:38:35] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:47:37] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36692.000000 [09:52:42] /sql on cassia is WARNING: DISK WARNING - free space: /sql 121900 MB (10% inode=99%): [10:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:07:43] Free Memory on damiana is WARNING: WARNING - 5.3% (441048 kB) free! [10:10:02] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36642.000000 [10:10:02] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37921.000000 [10:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36988.000000 [10:10:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36686.000000 [10:13:42] Free Memory on damiana is OK: OK - 7.1% (591780 kB) free. [10:15:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25217.000000 [10:16:02] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25236.000000 [10:16:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25273.000000 [10:16:43] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25273.000000 [10:26:53] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [10:26:53] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [10:27:02] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [10:27:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [10:27:52] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [10:28:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [10:28:42] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [10:33:42] Free Memory on damiana is WARNING: WARNING - 6.5% (547472 kB) free! [10:34:43] Free Memory on damiana is OK: OK - 7.2% (601576 kB) free. [10:36:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:36:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1267039.000000 [10:36:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:36:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 96093 MB (15% inode=99%): [10:37:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.01, 22.52, 22.83 [10:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:38:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39743.000000 [10:38:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 147739 MB (2% inode=52%): [10:38:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [10:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [10:38:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 26596.000000 [10:38:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:47:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40297.000000 [11:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 41531.000000 [11:10:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40252.000000 [11:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40588.000000 [11:10:42] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40286.000000 [11:11:42] Free Memory on damiana is WARNING: WARNING - 5.9% (496932 kB) free! [11:15:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28818.000000 [11:16:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28872.000000 [11:16:42] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28874.000000 [11:16:52] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28896.000000 [11:17:43] Free Memory on damiana is OK: OK - 7.1% (596420 kB) free. [11:26:52] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [11:26:52] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [11:27:02] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [11:27:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [11:28:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [11:28:42] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [11:28:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [11:47:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43900.000000 [12:06:51] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 45131.000000 [12:10:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43852.000000 [12:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44187.000000 [12:10:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43886.000000 [12:15:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32418.000000 [12:16:12] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 58357 MB (5% inode=99%): [12:16:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32473.000000 [12:16:42] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32473.000000 [12:16:52] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32495.000000 [12:26:52] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [12:26:52] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [12:27:02] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [12:27:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [12:28:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [12:28:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [12:28:42] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [12:33:42] Free Memory on damiana is WARNING: WARNING - 6.5% (545068 kB) free! [12:36:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:36:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1274240.000000 [12:36:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:36:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95605 MB (15% inode=99%): [12:37:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.96, 24.46, 24.24 [12:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:38:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46943.000000 [12:38:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 146699 MB (2% inode=52%): [12:38:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [12:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [12:38:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33796.000000 [12:38:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:42:42] Free Memory on damiana is OK: OK - 7.6% (637360 kB) free. [12:47:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47498.000000 [13:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:08:42] Free Memory on damiana is WARNING: WARNING - 6.4% (536568 kB) free! [13:09:42] Free Memory on damiana is OK: OK - 7.2% (604016 kB) free. [13:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48731.000000 [13:10:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47452.000000 [13:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47787.000000 [13:10:42] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 47487.000000 [13:16:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36073.000000 [13:16:43] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36074.000000 [13:16:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36077.000000 [13:16:51] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 36095.000000 [13:26:52] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [13:26:52] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [13:27:02] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [13:27:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [13:28:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [13:28:12] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 66148 MB (6% inode=99%): [13:28:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [13:28:42] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [13:33:42] Free Memory on damiana is WARNING: WARNING - 6.5% (542876 kB) free! [13:36:43] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:36:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1277841.000000 [13:36:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:36:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95329 MB (15% inode=99%): [13:37:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.86, 23.59, 23.74 [13:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:38:01] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50544.000000 [13:38:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 146215 MB (2% inode=52%): [13:38:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [13:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [13:38:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37396.000000 [13:38:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:40:42] Free Memory on damiana is OK: OK - 7.2% (602932 kB) free. [13:47:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51098.000000 [14:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:10:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51052.000000 [14:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52331.000000 [14:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51388.000000 [14:10:42] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51087.000000 [14:16:43] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39674.000000 [14:16:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39672.000000 [14:16:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39678.000000 [14:16:51] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 39695.000000 [14:19:47] hello all [14:26:52] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [14:26:53] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [14:27:02] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [14:27:05] DaB. * [Toolserver-l] Replication stopped [14:27:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [14:28:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [14:28:42] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [14:28:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [14:36:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:36:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1281440.000000 [14:36:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:36:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 95114 MB (15% inode=99%): [14:37:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.69, 23.11, 23.38 [14:37:06] @replag [14:37:07] liangent: s1-rr-a: 11h 21m 46s [+1.00 s/s]; s1-rr-a-wd: 14h 37m 57s [+1.00 s/s]; s1-user: 11h 21m 46s [+1.00 s/s]; s1-user-c: 11h 21m 47s [+1.00 s/s]; s1-user-wd: 15h 1m 6s [+1.00 s/s]; s2-user: 11h 21m 47s [+1.00 s/s]; s2-user-c: 2w 19h 57m 48s [+1.00 s/s]; s2-user-wd: 15h 1m 38s [+1.00 s/s] [14:37:08] liangent: s3-user: 11h 22m 13s [+1.00 s/s]; s4-rr-a: 11h 21m 47s [+1.00 s/s]; s4-user: 11h 21m 47s [+1.00 s/s]; s4-user-wd: 14h 43m 1s [+1.00 s/s]; s5-rr-a: 11h 21m 52s [+1.00 s/s]; s5-user: 11h 21m 52s [+1.00 s/s]; s5-user-c: 2w 19h 57m 48s [+1.00 s/s]; s5-user-wd: 15h 1m 38s [+1.00 s/s] [14:37:09] liangent: s6-user: 11h 22m 24s [+1.00 s/s]; s6-user-wd: 14h 59m 16s [+1.00 s/s]; s7-user: 11h 21m 49s [+1.00 s/s]; s7-user-wd: 14h 37m 57s [+1.00 s/s] [14:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:38:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54143.000000 [14:38:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [14:38:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 145686 MB (2% inode=52%): [14:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (3 errors): null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.CommunicationLost.desc:S3:oob:S4:2530:S9:ts-array5:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.descri [14:38:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 40995.000000 [14:38:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:45:42] Free Memory on damiana is WARNING: WARNING - 6.7% (561872 kB) free! [14:47:42] Free Memory on damiana is OK: OK - 7.1% (595240 kB) free. [14:47:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54699.000000 [14:50:08] DaBPunkt: gestern wuden einige neue Wiki erzeugt, die noch keine view auf dem ts haben [14:50:50] Merlissimo: die emails kamen als ich auf den weg ins bett war [14:52:04] Merlissimo: die datenbanken scheinen schon repliziert zu sein. Ich lege die views an wenn ich 'nen ruhigen Moment finde [14:52:19] ok, solange du das auf dem radar ist, ist alles prima - jira ticket konnte ich ja nicht erstellen [15:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:10:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54652.000000 [15:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55931.000000 [15:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54987.000000 [15:10:42] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54687.000000 [15:16:42] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43274.000000 [15:16:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43273.000000 [15:16:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43277.000000 [15:16:52] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43296.000000 [15:17:11] Merlissimo: ok, ich habe die views angelegt udn die haupt-kopie von toolserver.wiki geupdatet. namespaces und die kopien müssten sich in abshebarer Zeit selbst updaten [15:20:26] DaBPunkt: thx [15:26:25] DaBPunkt: no response at all??? [15:26:52] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [15:26:52] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [15:27:02] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [15:27:18] jeremyb_: their IRC was quite dead around noon. I got now some response and filled in a RT-ticket, which was at least moved :) [15:27:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [15:27:49] ok, good [15:28:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [15:28:42] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [15:28:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [15:29:22] DaBPunkt: also your mail footer has a pgp key id. you should make that longer. http://www.asheesh.org/note/debian/short-key-ids-are-bad-news.html [15:30:44] jeremyb_: I will take a look [15:33:42] Free Memory on damiana is WARNING: WARNING - 7.0% (587536 kB) free! [15:34:25] I see an Expired Key warning on your messages, too [15:34:30] although it may be a local problem [15:34:35] I didn't investigate it [15:34:43] Free Memory on damiana is OK: OK - 8.0% (670904 kB) free. [15:36:15] Platonides: that is a problem I faced everytiem I renew my key-pair. For some reasons the key-servers do not like to update keys where just the expiring-date changed [15:36:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:36:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1285040.000000 [15:36:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:36:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94855 MB (15% inode=99%): [15:37:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.61, 22.84, 23.00 [15:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:38:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57743.000000 [15:38:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 145099 MB (2% inode=52%): [15:38:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [15:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:38:43] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44595.000000 [15:38:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:47:43] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58299.000000 [15:49:29] jeremyb_: ok, should be fixed. Thanks for the notice [16:06:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:10:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58252.000000 [16:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 59531.000000 [16:10:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58588.000000 [16:10:43] Free Memory on damiana is WARNING: WARNING - 6.7% (565464 kB) free! [16:10:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58286.000000 [16:11:42] Free Memory on damiana is OK: OK - 7.1% (592468 kB) free. [16:16:42] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46874.000000 [16:16:52] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46895.000000 [16:17:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46932.000000 [16:17:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46937.000000 [16:26:52] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [16:26:53] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [16:27:03] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [16:27:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [16:28:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [16:28:43] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [16:28:43] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [16:36:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:36:43] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1288641.000000 [16:36:43] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:36:52] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94587 MB (15% inode=99%): [16:37:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.59, 22.79, 22.62 [16:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:38:01] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61345.000000 [16:38:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 144487 MB (2% inode=52%): [16:38:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [16:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:38:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:38:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 48195.000000 [16:47:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61898.000000 [17:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:10:11] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61852.000000 [17:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63131.000000 [17:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62188.000000 [17:11:42] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61946.000000 [17:16:42] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50474.000000 [17:16:52] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50495.000000 [17:17:43] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50532.000000 [17:17:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50538.000000 [17:26:52] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [17:26:52] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [17:27:02] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [17:27:42] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [17:28:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [17:28:42] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [17:28:43] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [17:36:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:36:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:36:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1292241.000000 [17:37:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94331 MB (15% inode=99%): [17:37:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.76, 22.38, 22.41 [17:37:11] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:38:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64944.000000 [17:38:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 143702 MB (2% inode=52%): [17:38:02] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [17:38:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [17:38:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 51795.000000 [17:38:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:47:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65499.000000 [17:58:01] @replag [17:58:02] Alchimista: s1-rr-a: 14h 42m 41s [+1.00 s/s]; s1-rr-a-wd: 17h 58m 52s [+1.00 s/s]; s1-user: 14h 42m 41s [+1.00 s/s]; s1-user-c: 14h 42m 42s [+1.00 s/s]; s1-user-wd: 18h 22m 1s [+1.00 s/s]; s2-user: 14h 42m 42s [+1.00 s/s]; s2-user-c: 2w 23h 18m 43s [+1.00 s/s]; s2-user-wd: 18h 22m 33s [+1.00 s/s] [17:58:03] Alchimista: s3-user: 14h 43m 8s [+1.00 s/s]; s4-rr-a: 14h 42m 42s [+1.00 s/s]; s4-user: 14h 42m 42s [+1.00 s/s]; s4-user-wd: 18h 3m 56s [+1.00 s/s]; s5-rr-a: 14h 42m 47s [+1.00 s/s]; s5-user: 14h 42m 47s [+1.00 s/s]; s5-user-c: 2w 23h 18m 43s [+1.00 s/s]; s5-user-wd: 18h 22m 33s [+1.00 s/s] [17:58:04] Alchimista: s6-user: 14h 43m 19s [+1.00 s/s]; s6-user-wd: 18h 20m 11s [+1.00 s/s]; s7-user: 14h 42m 44s [+1.00 s/s]; s7-user-wd: 17h 58m 52s [+1.00 s/s] [18:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66731.000000 [18:10:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65452.000000 [18:10:43] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65788.000000 [18:11:42] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 65547.000000 [18:16:42] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54074.000000 [18:17:02] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54097.000000 [18:17:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54132.000000 [18:17:43] Free Memory on damiana is WARNING: WARNING - 6.5% (542384 kB) free! [18:17:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54139.000000 [18:20:42] Free Memory on damiana is OK: OK - 7.3% (612016 kB) free. [18:27:02] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [18:27:02] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [18:27:03] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [18:28:03] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [18:28:43] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [18:28:43] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [18:28:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [18:36:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:36:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1295841.000000 [18:37:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 94125 MB (15% inode=99%): [18:37:03] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.52, 21.26, 21.34 [18:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:38:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 68544.000000 [18:38:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 143225 MB (2% inode=52%): [18:38:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [18:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:38:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 55396.000000 [18:38:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:40:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:47:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69099.000000 [18:57:42] Free Memory on damiana is WARNING: WARNING - 6.4% (533216 kB) free! [19:03:42] Free Memory on damiana is OK: OK - 7.2% (603836 kB) free. [19:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [19:10:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69052.000000 [19:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 70331.000000 [19:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69387.000000 [19:11:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 69146.000000 [19:16:43] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57673.000000 [19:17:02] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57697.000000 [19:17:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57732.000000 [19:17:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 57737.000000 [19:27:02] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [19:27:03] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [19:27:03] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [19:28:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [19:28:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [19:28:43] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [19:28:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [19:36:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:36:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1299441.000000 [19:37:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93821 MB (15% inode=99%): [19:37:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 21.08, 21.30, 21.46 [19:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:38:03] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72145.000000 [19:38:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 142573 MB (2% inode=52%): [19:38:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [19:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:38:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 58996.000000 [19:38:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:40:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:47:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72699.000000 [19:51:42] Free Memory on damiana is WARNING: WARNING - 6.7% (561104 kB) free! [19:55:42] Free Memory on damiana is OK: OK - 7.3% (613488 kB) free. [20:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 73932.000000 [20:10:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72653.000000 [20:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72988.000000 [20:11:43] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 72746.000000 [20:16:43] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61274.000000 [20:17:02] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61297.000000 [20:17:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61332.000000 [20:17:42] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61337.000000 [20:27:03] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [20:27:03] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [20:27:03] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [20:28:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [20:28:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [20:28:43] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [20:28:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [20:36:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:36:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1303040.000000 [20:37:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93492 MB (15% inode=99%): [20:37:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 23.13, 23.35, 22.69 [20:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:38:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 75745.000000 [20:38:02] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 141865 MB (2% inode=52%): [20:38:02] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [20:38:02] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [20:38:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62595.000000 [20:38:43] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:40:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:47:42] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76298.000000 [21:06:42] Free Memory on damiana is WARNING: WARNING - 5.9% (498292 kB) free! [21:06:52] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:09:43] Free Memory on damiana is OK: OK - 7.8% (654940 kB) free. [21:10:12] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76253.000000 [21:10:12] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 77531.000000 [21:10:42] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76590.000000 [21:11:42] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 76348.000000 [21:16:42] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64876.000000 [21:17:02] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64897.000000 [21:17:42] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64933.000000 [21:17:43] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 64939.000000 [21:27:02] MySQL slave on thyme is CRITICAL: (Return code of 139 is out of bounds) [21:27:03] MySQL slave on daphne is CRITICAL: (Return code of 139 is out of bounds) [21:27:03] MySQL slave on cassia is CRITICAL: (Return code of 139 is out of bounds) [21:28:02] MySQL slave on z-dat-s3-a is CRITICAL: (Return code of 139 is out of bounds) [21:28:42] MySQL slave on z-dat-s7-a is CRITICAL: (Return code of 139 is out of bounds) [21:28:43] MySQL slave on z-dat-s6-a is CRITICAL: (Return code of 139 is out of bounds) [21:28:43] MySQL slave on z-dat-s4-a is CRITICAL: (Return code of 139 is out of bounds) [21:36:42] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:36:42] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1306640.000000 [21:37:02] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 93171 MB (15% inode=99%): [21:37:02] Load avg. on willow is CRITICAL: CRITICAL - load average: 24.64, 24.55, 23.97 [21:37:12] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:38:02] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79346.000000 [21:38:03] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 140859 MB (2% inode=52%): [21:38:03] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [21:38:03] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:38:42] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66196.000000 [21:38:42] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:40:42] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:47:45] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79899.000000 [21:49:45] Free Memory on damiana is WARNING: WARNING - 6.7% (563972 kB) free! [21:52:45] /sql on cassia is WARNING: DISK WARNING - free space: /sql 125357 MB (10% inode=99%): [21:52:46] Free Memory on damiana is OK: OK - 7.4% (618208 kB) free. [22:02:45] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:04:11] hi DaBPunkt [22:04:26] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 242 bytes in 0.536 second response time [22:04:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:05:04] Maarten Dammers * Re: [Toolserver-l] Replication stopped [22:05:15] @replag [22:05:15] multichill: s1-rr-a: 18h 49m 54s [+1.00 s/s]; s1-rr-a-wd: 22h 6m 5s [+1.00 s/s]; s1-user: 18h 49m 54s [+1.00 s/s]; s1-user-c: 18h 49m 55s [+1.00 s/s]; s1-user-wd: 22h 29m 14s [+1.00 s/s]; s2-user: 18h 49m 55s [+1.00 s/s]; s2-user-c: 2w 1d 3h 25m 56s [+1.00 s/s]; s2-user-wd: 22h 29m 46s [+1.00 s/s] [22:05:16] multichill: s3-user: 18h 50m 21s [+1.00 s/s]; s4-rr-a: 18h 49m 55s [+1.00 s/s]; s4-user: 18h 49m 55s [+1.00 s/s]; s4-user-wd: 22h 11m 9s [+1.00 s/s]; s5-rr-a: 18h 50m 0s [+1.00 s/s]; s5-user: 18h 50m 0s [+1.00 s/s]; s5-user-c: 2w 1d 3h 25m 56s [+1.00 s/s]; s5-user-wd: 22h 29m 46s [+1.00 s/s] [22:05:17] multichill: s6-user: 18h 50m 32s [+1.00 s/s]; s6-user-wd: 22h 27m 24s [+1.00 s/s]; s7-user: 18h 49m 57s [+1.00 s/s]; s7-user-wd: 22h 6m 5s [+1.00 s/s] [22:05:34] ottomata: So no, no replication yet. I think that has to manually restarted [22:06:08] ok, DaBPunkt, I don't have login on amaranth, so I can't fix anything beyond getting it to boot [22:06:52] ottomata: ok, thank you very much [22:07:01] I will look what is the problem from here [22:07:35] great! [22:09:41] ottomata: Thanks for helping! [22:09:50] yup, glad its back up [22:09:55] laters all! [22:10:07] Time for bed now, I'm also on standby and I think I'm going to be lucky tonight (not) [22:10:25] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 81149.000000 [22:10:26] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79871.000000 [22:10:26] multichill: toi, toi, toi! [22:10:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 80190.000000 [22:11:29] ok, the sql-tunnels are working. I will be back in a moment [22:11:44] @replag [22:11:44] multichill: s1-rr-a: 18h 53m 50s [+0.61 s/s]; s1-rr-a-wd: 22h 12m 35s [+1.00 s/s]; s1-user: 18h 55m 54s [+0.92 s/s]; s1-user-c: 18h 53m 54s [+0.61 s/s]; s1-user-wd: 22h 35m 44s [+1.00 s/s]; s2-user: 18h 54m 18s [+0.68 s/s]; s2-user-c: 2w 1d 3h 32m 26s [+1.00 s/s]; s2-user-wd: 22h 36m 16s [+1.00 s/s] [22:11:46] multichill: s3-user: 18h 54m 10s [+0.59 s/s]; s4-rr-a: 18h 39m 12s [-1.65 s/s]; s4-user: 18h 39m 12s [-1.65 s/s]; s4-user-wd: 22h 17m 39s [+1.00 s/s]; s5-rr-a: 17h 43m 12s [-10.29 s/s]; s5-user: 17h 43m 12s [-10.29 s/s]; s5-user-c: 2w 1d 3h 32m 26s [+1.00 s/s]; s5-user-wd: 22h 36m 16s [+1.00 s/s] [22:11:46] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 79948.000000 [22:11:48] multichill: s6-user: 18h 30m 51s [-3.03 s/s]; s6-user-wd: 22h 33m 54s [+1.00 s/s]; s7-user: 18h 43m 3s [-1.06 s/s]; s7-user-wd: 22h 12m 35s [+1.00 s/s] [22:11:53] :-) [22:16:46] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 62485.000000 [22:17:26] s5 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 54864.000000 [22:17:45] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67468.000000 [22:17:45] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 66848.000000 [22:19:04] DaB. * Re: [Toolserver-l] Replication stopped [22:24:44] @replag [22:24:45] russblau: s1-rr-a: 18h 33m 58s [-2.70 s/s]; s1-rr-a-wd: 22h 24m 56s [+0.90 s/s]; s1-user: 18h 50m 8s [+0.81 s/s]; s1-user-c: 17h 48m 20s [-2.51 s/s]; s1-user-wd: 22h 47m 55s [+0.78 s/s]; s2-user: 18h 36m 34s [-0.53 s/s]; s2-user-c: 2w 1d 3h 44m 26s [+0.90 s/s]; s2-user-wd: 22h 48m 35s [+0.96 s/s] [22:24:46] russblau: s3-user: 18h 13m 57s [-1.99 s/s]; s4-rr-a: 15h 40s [-8.57 s/s]; s4-user: 15h 40s [-8.57 s/s]; s4-user-wd: 22h 30m 1s [+0.96 s/s]; s5-rr-a: 13h 57m 31s [-12.29 s/s]; s5-user: 13h 57m 31s [-12.30 s/s]; s5-user-c: 2w 1d 3h 44m 26s [+0.90 s/s]; s5-user-wd: 22h 48m 35s [+0.96 s/s] [22:24:46] russblau: s6-user: 16h 19m 8s [-5.19 s/s]; s6-user-wd: 22h 46m 4s [+0.87 s/s]; s7-user: 16h 58m 17s [-7.79 s/s]; s7-user-wd: 22h 24m 56s [+0.90 s/s] [22:27:25] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 66357 [22:27:25] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 52102 [22:27:26] MySQL slave on cassia is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 48693 [22:28:26] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 64405 [22:28:45] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57675 [22:28:46] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 57184 [22:28:46] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 53218 [22:30:01] [[Special:Log/newusers]] create 10 * Numeracy * (New user account) [22:31:11] [[User talk:Numeracy]] !N 10https://wiki.toolserver.org/w/index.php?oldid=7792&rcid=21492 * Numeracy * (+1184) (Created page with "Young ones who have just started school might need a tutorial or books which can aid them understand their courses promptly without having difficulties. Studies may appear hard t...") [22:31:37] I hate spam… [22:31:45] [[Special:Log/delete]] delete 10 * Dab * (deleted "[[02User talk:Numeracy10]]": SPAM) [22:32:02] [[Special:Log/block]] block 10 * Dab * (blocked [[02User:Numeracy10]] with an expiry time of infinite (account creation disabled): Spam) [22:32:52] mm, looks like jira finally broke… [22:36:46] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:36:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:36:54] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1310068.000000 [22:37:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:37:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92895 MB (15% inode=99%): [22:37:25] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.34, 22.84, 23.17 [22:38:25] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 139482 MB (2% inode=52%): [22:38:26] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82858.000000 [22:38:34] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 67142 [22:38:35] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:38:45] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 67128.000000 [22:38:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:39:06] @replag [22:39:06] russblau: s1-rr-a: 17h 48m 1s [-3.20 s/s]; s1-rr-a-wd: 22h 38m 23s [+0.94 s/s]; s1-user: 18h 38m 3s [-0.84 s/s]; s1-user-c: 16h 1m 8s [-7.47 s/s]; s1-user-wd: 23h 58s [+0.91 s/s]; s2-user: 17h 48m 18s [-3.36 s/s]; s2-user-c: 2w 1d 3h 56m 10s [+0.82 s/s]; s2-user-wd: 23h 1m 38s [+0.91 s/s] [22:39:07] russblau: s3-user: 17h 8m 2s [-4.59 s/s]; s4-rr-a: 12h 17m 50s [-11.34 s/s]; s4-user: 12h 17m 50s [-11.34 s/s]; s4-user-wd: 22h 43m 23s [+0.93 s/s]; s5-rr-a: 10h 14m 24s [-15.54 s/s]; s5-user: 10h 14m 24s [-15.54 s/s]; s5-user-c: 2w 1d 3h 56m 10s [+0.82 s/s]; s5-user-wd: 23h 1m 38s [+0.91 s/s] [22:39:08] russblau: s6-user: 14h 9m 30s [-9.03 s/s]; s6-user-wd: 22h 59m 0s [+0.90 s/s]; s7-user: 14h 34m 24s [-10.02 s/s]; s7-user-wd: 22h 38m 23s [+0.94 s/s] [22:40:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:47:55] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83154.000000 [23:02:55] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:04:25] MySQL slave on cassia is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1875 [23:04:25] s5 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1873.000000 [23:04:35] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:04:35] jira.toolserver.org on web.amaranth is WARNING: HTTP WARNING: HTTP/1.1 302 Moved Temporarily - 243 bytes in 0.525 second response time [23:05:26] MySQL slave on cassia is OK: Uptime: 4759885 Threads: 21 Questions: 12213101728 Slow queries: 771785 Opens: 17843246 Flush tables: 1 Open tables: 16044 Queries per second avg: 2565.839 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [23:05:26] s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1.000000 [23:10:25] wikidata replag on z-dat-s6-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 83818.000000 [23:10:45] wikidata replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82883.000000 [23:11:25] wikidata replag on z-dat-s7-a is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82574.000000 [23:11:45] wikidata replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 82574.000000 [23:16:45] s4 replag on daphne is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 20416.000000 [23:17:45] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 52548.000000 [23:17:45] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 50023.000000 [23:27:26] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 50005 [23:27:26] MySQL slave on daphne is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14170 [23:28:45] MySQL slave on z-dat-s7-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 32118 [23:28:46] MySQL slave on z-dat-s6-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 45184 [23:28:46] MySQL slave on z-dat-s4-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19208 [23:29:25] MySQL slave on z-dat-s3-a is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 44547 [23:36:45] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:36:46] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:36:55] s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 1310561.000000 [23:37:25] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:37:25] Load avg. on willow is CRITICAL: CRITICAL - load average: 22.57, 23.45, 23.78 [23:37:25] /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 92635 MB (15% inode=99%): [23:38:26] wikidata replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 84875.000000 [23:38:26] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 140159 MB (2% inode=52%): [23:38:35] MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 61370 [23:38:35] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [23:38:46] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 61356.000000 [23:38:46] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:40:55] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:43:25] MySQL slave on daphne is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3352 [23:43:45] s4 replag on daphne is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3177.000000 [23:45:45] s4 replag on daphne is OK: QUERY OK: SELECT ts_rc_age() returned 1573.000000 [23:46:25] MySQL slave on daphne is OK: Uptime: 3462676 Threads: 17 Questions: 788532557 Slow queries: 40948 Opens: 155535 Flush tables: 1 Open tables: 1275 Queries per second avg: 227.723 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1033 [23:46:45] Free Memory on damiana is WARNING: WARNING - 6.2% (517948 kB) free! [23:47:54] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 85047.000000 [23:53:45] Free Memory on damiana is OK: OK - 7.1% (597764 kB) free. [23:57:45] MySQL slave on z-dat-s4-a is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3146 [23:59:45] MySQL slave on z-dat-s4-a is OK: Uptime: 1990045 Threads: 1 Questions: 252398384 Slow queries: 72 Opens: 391 Flush tables: 7 Open tables: 119 Queries per second avg: 126.830 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1558