[00:22:16] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 357373 MB (6% inode=68%): [00:23:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [00:23:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:24:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:24:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:27:07] [[Account approval process/for reviewers]] 10https://wiki.toolserver.org/w/index.php?diff=7814&oldid=7431&rcid=21558 * Dab * (+173) (/* Question-Text */ mail to OTRS) [00:27:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:30:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43729 MB (7% inode=99%): [00:53:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:15:17] Load avg. on ortelius is WARNING: WARNING - load average: 21.77, 18.52, 10.49 [01:22:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 359936 MB (6% inode=68%): [01:22:22] @replag [01:22:23] DaBPunkt: s1-rr-a-wd: 1m 21s [+0.00 s/s]; s1-user-wd: 1m 22s [+0.00 s/s]; s2-rr: 31m 42s [+0.01 s/s]; s2-user: 31m 42s [+0.01 s/s]; s2-user-c: error; s2-user-wd: 24m 0s [+0.03 s/s]; s3-user: 1m 38s [+0.00 s/s]; s4-user-wd: 1m 22s [+0.00 s/s] [01:22:24] DaBPunkt: s5-user-c: 1m 10s [-0.00 s/s]; s5-user-wd: 1m 21s [+0.00 s/s]; s6-user: 11s [-0.00 s/s]; s6-user-wd: 1m 22s [+0.00 s/s]; s7-user-wd: 1m 21s [+0.00 s/s] [01:23:05] nacht ts [01:23:17] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [01:23:28] @replag [01:23:28] DaBPunkt: s1-rr-a-wd: 1m 33s [+0.18 s/s]; s1-user-wd: 1m 33s [+0.17 s/s]; s2-rr: 32m 1s [+0.29 s/s]; s2-user: 32m 1s [+0.29 s/s]; s2-user-c: error; s2-user-wd: 23m 16s [-0.67 s/s]; s3-user: 40s [-0.89 s/s]; s4-user-wd: 1m 33s [+0.17 s/s] [01:23:29] DaBPunkt: s5-user-c: 1m 42s [+0.49 s/s]; s5-user-wd: 1m 33s [+0.18 s/s]; s6-user-wd: 1m 35s [+0.20 s/s]; s7-user-wd: 1m 31s [+0.15 s/s] [01:23:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:24:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:24:17] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1923 [01:24:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:24:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:27:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:28:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:30:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43355 MB (7% inode=99%): [01:33:17] MySQL slave on z-dat-s2-b is OK: Uptime: 281192 Threads: 10 Questions: 201923303 Slow queries: 4019 Opens: 9783309 Flush tables: 1 Open tables: 64 Queries per second avg: 718.97 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1704 [01:38:27] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70340 MB (7% inode=99%): [01:44:17] Load avg. on ortelius is CRITICAL: CRITICAL - load average: 31.48, 24.04, 20.00 [01:53:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [01:56:17] Load avg. on ortelius is WARNING: WARNING - load average: 8.21, 18.50, 20.00 [02:02:17] Load avg. on ortelius is OK: OK - load average: 4.50, 8.02, 14.51 [02:22:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 359365 MB (6% inode=68%): [02:23:16] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [02:23:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:24:07] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:24:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:24:58] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:27:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:28:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:30:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 43508 MB (7% inode=99%): [02:53:17] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [03:07:07] Load avg. on willow is WARNING: WARNING - load average: 14.21, 15.38, 15.00 [03:09:07] Load avg. on willow is OK: OK - load average: 13.30, 14.71, 14.80 [03:17:13] Load avg. on willow is WARNING: WARNING - load average: 14.96, 15.68, 15.34 [03:22:17] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 358724 MB (6% inode=68%): [03:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [03:23:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:24:16] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:24:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:24:57] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:27:57] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:28:57] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:30:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42984 MB (7% inode=99%): [03:30:08] Load avg. on willow is OK: OK - load average: 13.88, 14.62, 14.98 [04:17:08] Load avg. on willow is WARNING: WARNING - load average: 15.72, 16.45, 16.43 [04:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 359651 MB (6% inode=68%): [04:23:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [04:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [04:23:36] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:24:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:24:57] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:24:58] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:28:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:28:58] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42776 MB (6% inode=99%): [04:43:07] Load avg. on willow is OK: OK - load average: 13.97, 14.38, 14.98 [05:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 358864 MB (6% inode=68%): [05:23:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [05:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [05:23:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:24:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:25:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:25:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:29:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:31:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42654 MB (6% inode=99%): [05:36:08] Load avg. on willow is WARNING: WARNING - load average: 15.56, 15.72, 15.31 [05:39:07] Load avg. on willow is OK: OK - load average: 13.73, 14.72, 14.98 [06:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 356236 MB (6% inode=68%): [06:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [06:23:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:25:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:25:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:29:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42353 MB (6% inode=99%): [06:52:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [07:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 356028 MB (6% inode=68%): [07:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [07:23:37] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:25:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:25:17] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42217 MB (6% inode=99%): [07:33:27] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2073 [07:52:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [08:02:27] MySQL slave on z-dat-s2-b is OK: Uptime: 304539 Threads: 10 Questions: 219288410 Slow queries: 4405 Opens: 10498407 Flush tables: 1 Open tables: 64 Queries per second avg: 720.66 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1683 [08:17:07] Load avg. on willow is WARNING: WARNING - load average: 14.64, 15.21, 14.68 [08:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 355578 MB (6% inode=68%): [08:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [08:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:25:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:28:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 42022 MB (6% inode=99%): [08:52:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [09:02:07] MySQL slave on rosemary is CRITICAL: (Return code of 139 is out of bounds) [09:03:08] MySQL slave on rosemary is OK: Uptime: 580876 Threads: 14 Questions: 383399534 Slow queries: 113156 Opens: 42766 Flush tables: 1 Open tables: 3103 Queries per second avg: 660.36 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 411 [09:12:50] @replag [09:12:50] Merlissimo: s1-rr-a-wd: 1m 25s [-0.00 s/s]; s1-user-wd: 1m 24s [-0.00 s/s]; s2-user-c: error; s2-user-wd: 1m 24s [-0.05 s/s]; s4-user-wd: 1m 25s [-0.00 s/s]; s5-user-c: 1m 33s [-0.00 s/s]; s5-user-wd: 1m 24s [-0.00 s/s]; s6-user-wd: 1m 25s [-0.00 s/s] [09:12:51] Merlissimo: s7-user-wd: 1m 25s [-0.00 s/s] [09:13:54] @replag all [09:13:55] Merlissimo: s1-rr-a: 1s [-0.00 s/s]; s1-rr-a-wd: 1m 36s [+0.17 s/s]; s1-user: 1s [-0.00 s/s]; s1-user-c: 1s [-0.00 s/s]; s1-user-wd: 1m 28s [+0.06 s/s]; s2-rr: 16s [-0.07 s/s]; s2-user: 16s [-0.07 s/s]; s2-user-c: error [09:13:56] Merlissimo: s2-user-wd: 1m 28s [+0.06 s/s]; s3-user: 4s [-0.00 s/s]; s4-rr-a: 1s [-0.00 s/s]; s4-user: 1s [-0.00 s/s]; s4-user-wd: 1m 36s [+0.17 s/s]; s5-rr-a: 1s [-0.00 s/s]; s5-user: 1s [-0.00 s/s]; s5-user-c: 1m 53s [+0.31 s/s] [09:13:57] Merlissimo: s5-user-wd: 1m 31s [+0.11 s/s]; s6-user: 2s [-0.00 s/s]; s6-user-wd: 1m 36s [+0.17 s/s]; s7-user: 1s [-0.00 s/s]; s7-user-wd: 1m 28s [+0.05 s/s] [09:17:07] Load avg. on willow is WARNING: WARNING - load average: 15.38, 16.13, 16.04 [09:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 355171 MB (6% inode=68%): [09:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [09:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:25:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:31:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41787 MB (6% inode=99%): [09:52:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [10:17:08] Load avg. on willow is WARNING: WARNING - load average: 14.91, 15.77, 15.73 [10:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 354360 MB (6% inode=68%): [10:23:28] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [10:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:28:07] Load avg. on willow is OK: OK - load average: 14.07, 14.47, 15.00 [10:29:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41611 MB (6% inode=99%): [10:37:07] Load avg. on willow is WARNING: WARNING - load average: 14.72, 15.12, 15.16 [10:40:07] Load avg. on willow is OK: OK - load average: 14.33, 14.71, 14.99 [10:44:47] wikidata replag on z-dat-s2-b is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1886.000000 [10:48:28] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1907 [10:50:47] wikidata replag on z-dat-s2-b is OK: QUERY OK: SELECT ts_rc_age() returned 1789.000000 [10:51:27] MySQL slave on z-dat-s2-b is OK: Uptime: 314680 Threads: 8 Questions: 230524056 Slow queries: 4615 Opens: 10822099 Flush tables: 1 Open tables: 64 Queries per second avg: 732.566 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1615 [10:52:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [11:19:07] Load avg. on willow is WARNING: WARNING - load average: 14.70, 15.70, 15.76 [11:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353810 MB (6% inode=68%): [11:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [11:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:25:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41375 MB (6% inode=99%): [11:52:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [12:09:07] s4 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [12:09:07] s1 replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [12:09:07] MySQL on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [12:09:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on rosemary (146) [12:09:07] MySQL slave on rosemary is CRITICAL: Cant connect to MySQL server on rosemary (146) [12:09:17] SSH on rosemary is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:09:38] Load avg. on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:09:47] Environment IPMI on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:10:17] RAID on rosemary is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:13:47] SMTP on rosemary is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:13:57] RAID on rosemary is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [12:14:07] SSH on rosemary is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0) [12:14:07] Load avg. on rosemary is OK: OK - load average: 0.41, 0.31, 0.55 [12:14:17] Environment IPMI on rosemary is OK: ok: temperature ok fan ok voltage ok chassis ok [12:14:36] SMTP on rosemary is OK: SMTP OK - 0.013 sec. response time [12:19:07] Load avg. on willow is WARNING: WARNING - load average: 16.01, 16.91, 16.67 [12:22:28] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353422 MB (6% inode=68%): [12:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [12:23:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 41141 MB (6% inode=99%): [12:37:08] MySQL on rosemary is OK: Uptime: 1392 Threads: 48 Questions: 1985 Slow queries: 15 Opens: 159 Flush tables: 1 Open tables: 149 Queries per second avg: 1.426 [12:37:08] MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2062 [12:37:17] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2142.000000 [12:37:27] s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2054.000000 [12:37:27] s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2059.000000 [12:40:07] s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1552.000000 [12:42:07] s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1776.000000 [12:42:07] MySQL slave on rosemary is OK: Uptime: 1694 Threads: 11 Questions: 199152 Slow queries: 208 Opens: 423 Flush tables: 1 Open tables: 399 Queries per second avg: 117.563 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1766 [12:53:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [13:02:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3641.000000 [13:19:07] Load avg. on willow is WARNING: WARNING - load average: 14.63, 15.32, 15.70 [13:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 353038 MB (6% inode=68%): [13:23:26] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [13:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:31:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40913 MB (6% inode=99%): [13:38:37] /sql on rosemary is WARNING: DISK WARNING - free space: /sql 70324 MB (7% inode=99%): [13:40:28] hello all [13:40:32] @replag [13:40:32] DaBPunkt: s1-rr-a-wd: 1m 14s [-0.00 s/s]; s1-user: 2m 43s [+0.01 s/s]; s1-user-c: 3m 27s [+0.01 s/s]; s1-user-wd: 1h 39m 9s [+0.37 s/s]; s2-rr: 23m 28s [+0.09 s/s]; s2-user: 23m 28s [+0.09 s/s]; s2-user-c: error; s2-user-wd: 1m 14s [-0.00 s/s] [13:40:33] DaBPunkt: s3-user: 15s [+0.00 s/s]; s4-user-wd: 1m 14s [-0.00 s/s]; s5-user-wd: 1m 14s [-0.00 s/s]; s6-user-wd: 1m 14s [-0.00 s/s]; s7-user-wd: 1m 14s [-0.00 s/s] [14:02:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7243.000000 [14:19:07] Load avg. on willow is WARNING: WARNING - load average: 14.95, 15.79, 16.12 [14:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 352617 MB (6% inode=68%): [14:22:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [14:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [14:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:29:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40673 MB (6% inode=99%): [15:02:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10842.000000 [15:19:07] Load avg. on willow is WARNING: WARNING - load average: 15.18, 16.07, 16.28 [15:22:33] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 352088 MB (6% inode=68%): [15:22:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [15:23:26] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [15:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:31:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40419 MB (6% inode=99%): [16:02:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14443.000000 [16:19:07] Load avg. on willow is WARNING: WARNING - load average: 15.80, 16.24, 16.48 [16:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 351373 MB (6% inode=68%): [16:22:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [16:23:26] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [16:23:48] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:25:24] * jeremyb_ waves [16:25:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:25:34] Silke_WMDE_: DaBPunkt: amaranth bump? [16:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:29:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 40200 MB (6% inode=99%): [16:45:01] jeremyb_: what? [16:45:27] Silke_WMDE_: amaranth. tampa host [16:45:46] RT 4497 [16:53:38] I don't have access to RT [16:53:49] I suppose it's the RAM thing? [17:02:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18043.000000 [17:02:46] Silke_WMDE_: get access to RT :) [17:02:54] * DanielK_WMDE has lost his password [17:03:38] tststs [17:19:07] Load avg. on willow is WARNING: WARNING - load average: 15.67, 16.14, 16.36 [17:22:16] Silke_WMDE_: you should have access. just do a password reset [17:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 350278 MB (6% inode=68%): [17:22:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [17:23:27] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [17:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:23:53] Silke_WMDE_: in any case the point was the ticket is waiting on WMDE / TS-admins not waiting on WMF [17:24:30] the account to use for password reset would be silke.meyer@wikimedia.de [17:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:28:08] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:31:08] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39977 MB (6% inode=99%): [17:46:06] Silke Meyer * [Toolserver-l] List of tools on the toolserver - Help needed [17:57:32] @replag [17:57:33] Alchimista: s1-rr-a-wd: 1m 14s [-]; s1-user: 46s [-0.01 s/s]; s1-user-wd: 5h 56m 10s [+1.00 s/s]; s2-rr: 2m 50s [-0.08 s/s]; s2-user: 2m 50s [-0.08 s/s]; s2-user-c: error; s2-user-wd: 2m 41s [+0.01 s/s]; s4-user-wd: 1m 14s [-] [17:57:34] Alchimista: s5-user-c: 1m 5s [-0.00 s/s]; s5-user-wd: 1m 13s [-0.00 s/s]; s6-user-wd: 1m 13s [-0.00 s/s]; s7-user-wd: 1m 12s [-0.00 s/s] [17:59:25] jeremyb_: thanks, I'm in. Yes, there was a follow-up via e-mail. [18:00:07] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [18:02:08] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21642.000000 [18:19:08] Load avg. on willow is WARNING: WARNING - load average: 16.93, 17.52, 17.77 [18:22:26] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 349663 MB (6% inode=68%): [18:23:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [18:23:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [18:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:25:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39705 MB (6% inode=99%): [18:55:57] DaBPunkt: is qlogin a valid use of the new sge rule? [19:00:07] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:02:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 25242.000000 [19:04:42] Coren: are you around? [19:04:58] Betacommand: On and off. What's up? [19:05:31] Coren: see PM [19:19:07] Load avg. on willow is WARNING: WARNING - load average: 15.86, 17.13, 17.76 [19:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 349179 MB (6% inode=68%): [19:23:26] giftpflanze: pardon? [19:23:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [19:23:39] mh [19:23:50] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39411 MB (6% inode=99%): [19:38:26] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2087 [19:52:56] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [20:00:08] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [20:02:13] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28842.000000 [20:08:15] anyone knows if revision table has edits delected and or hidden? [20:14:26] Alchimista: deleted: yes, hidden: no [20:17:58] i'm making some querys, to a research, but the resoults aren't consistent. "SELECT COUNT(*) FROM revision WHERE rev_user = 0" -> total ip's edit [20:18:13] SELECT /* SLOW_OK LIMIT:180 */ COUNT(*) FROM revision JOIN user_groups ON rev_user = ug_user WHERE rev_user > 0 AND ug_group != 'bot'; -> registred users, non bot [20:18:34] SELECT /* SLOW_OK LIMIT:180 */ COUNT(*) FROM revision JOIN user_groups ON rev_user = ug_user WHERE ug_group = 'bot'; -> total bot edits [20:18:51] DaBPunkt: are this querys ok? ^ [20:19:07] Load avg. on willow is WARNING: WARNING - load average: 15.88, 16.52, 16.79 [20:20:28] Alchimista: they fail for bots which have any other right beside "bot" [20:22:27] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 348664 MB (6% inode=68%): [20:23:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [20:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:25:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:29:07] Load avg. on willow is OK: OK - load average: 4.72, 11.89, 14.88 [20:29:38] half of a dozen python-bots later… [20:30:49] DaBPunkt: dammit, it never crossed my mind :s how do i set it to count all bots edits? d i have to use a subquery? My sql is not that good [20:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 39084 MB (6% inode=99%): [20:31:41] Alchimista: that should be possible with your query [20:31:56] only the counting of NOT is tricky [20:32:37] DaBPunkt: making the sum of those 3 querys, the value doesn't match to "SELECT /* SLOW_OK LIMIT:180 */ COUNT(*) FROM revision;" [20:33:06] and it's *just* a diference of about 5176717 [20:33:44] too much or too little? [20:33:49] too little [20:35:18] and still, the total diference for the total edits given by pt.wp special page, is about 10 000 000 [20:35:21] I see the problem. The second query is also wrong [20:36:15] normal users do not have an entry in the ug-table# [20:36:27] and so the query does not find them [20:37:23] ah, so those without any flag aren't there? [20:38:17] yes [20:38:21] For example: [20:38:28] SELECT /* SLOW_OK LIMIT:180 */ COUNT(*) FROM revision JOIN user_groups ON rev_user = ug_user WHERE rev_user > 0 AND ug_group != 'bot' AND rev_user=41429 LIMIT 1; [20:38:37] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2629 [20:38:41] Says that I have no edits [20:38:45] (in enwp) [20:38:50] but I have [20:39:32] got it , so i must get another way to filter non bot edits [20:44:19] Alchimista: putting the bot-user-ids in a temp-table and than left-join with that table and using IS NULL should work [20:46:35] DaBPunkt: you've opened a window to new horizons :P Thanks for the help [20:46:52] np [20:53:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [21:00:07] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [21:02:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 32443.000000 [21:19:43] @replag [21:19:43] DaBPunkt: s1-rr-a-wd: 55s [-0.00 s/s]; s1-user-wd: 9h 18m 20s [+1.00 s/s]; s2-rr: 40m 44s [+0.19 s/s]; s2-user: 40m 44s [+0.19 s/s]; s2-user-c: error; s2-user-wd: 1m 4s [-0.01 s/s]; s4-user-wd: 55s [-0.00 s/s]; s5-user-c: 35s [-0.00 s/s] [21:19:44] DaBPunkt: s5-user-wd: 55s [-0.00 s/s]; s6-user-wd: 1m 1s [-0.00 s/s]; s7-user-wd: 1m 1s [-0.00 s/s] [21:22:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 348260 MB (6% inode=68%): [21:23:36] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [21:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:25:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:26:33] @replag [21:26:33] multichill: s1-rr-a-wd: 58s [+0.01 s/s]; s1-user-wd: 8h 43m 1s [-5.16 s/s]; s2-rr: 45m 23s [+0.68 s/s]; s2-user: 45m 23s [+0.68 s/s]; s2-user-c: error; s2-user-wd: 3m 42s [+0.39 s/s]; s4-user-wd: 1m 1s [+0.01 s/s]; s5-user-c: 25s [-0.02 s/s] [21:26:34] multichill: s5-user-wd: 58s [+0.01 s/s]; s6-user-wd: 1m 3s [+0.00 s/s]; s7-user-wd: 1m 3s [+0.00 s/s] [21:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:29:08] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 38808 MB (6% inode=99%): [21:38:37] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2316 [21:45:37] MySQL slave on z-dat-s2-b is OK: Uptime: 353928 Threads: 16 Questions: 256782581 Slow queries: 5271 Opens: 12235046 Flush tables: 1 Open tables: 64 Queries per second avg: 725.522 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1766 [22:00:07] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [22:02:07] wikidata replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 18903.000000 [22:08:12] I need something tested in IE10. Anyone running Windows 8? [22:22:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 347343 MB (6% inode=68%): [22:22:57] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [22:23:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors, 1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A:, null :OSGi.com.sun.storage.cam.agent(com.sun.netstorage.fm.storade.agent.Messages):monitor.Communicatio [22:23:46] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:25:07] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:25:08] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:25:26] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:27:37] MySQL slave on z-dat-s2-b is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1872 [22:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 38484 MB (6% inode=99%): [22:49:37] MySQL slave on z-dat-s2-b is OK: Uptime: 357769 Threads: 11 Questions: 257971549 Slow queries: 5353 Opens: 12393928 Flush tables: 1 Open tables: 65 Queries per second avg: 721.56 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1661 [23:00:07] APT on nightshade is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [23:00:08] wikidata replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3600.000000 [23:12:07] wikidata replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1789.000000 [23:15:07] APT on nightshade is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:22:37] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 320401 MB (6% inode=68%): [23:23:07] Virtual disks on far1-n1-oe16-esams.mgmt is CRITICAL: OK 3, WARN 0, CRIT 1: far1-n1-fast3 FTOL, far1-n1-bulk CRIT, far1-n1-fast2 FTOL, far1-n1-fast FTOL [23:23:37] CAM on hemlock is CRITICAL: CRITICAL - Storage ts-array5 (2 errors): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.B:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_EXPIRED_BATTERY.description:S17:Tray.85.Battery.A: [23:23:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:24:37] /mnt user-store on rosemary is CRITICAL: DISK CRITICAL - free space: /mnt 318853 MB (5% inode=68%): [23:25:07] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:25:08] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:25:27] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:28:07] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:28:36] /mnt user-store on rosemary is WARNING: DISK WARNING - free space: /mnt 322276 MB (6% inode=68%): [23:29:07] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:31:07] /sql on ptolemy is CRITICAL: DISK CRITICAL - free space: /sql 38207 MB (6% inode=99%):