[00:13:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [00:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:25:03] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [00:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [00:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [00:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [00:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [00:42:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [00:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [01:13:23] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [01:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [01:28:43] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [01:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [01:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [01:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [01:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [01:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [02:13:23] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [02:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [02:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [02:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [02:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [02:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [02:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [02:43:13] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:03:24] Environment IPMI on wolfsbane is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:04:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:24:13] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [03:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:24:44] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:25:04] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [03:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [03:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [03:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [03:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [03:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [03:43:14] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [03:53:54] /sql on z-dat-s3-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:53:54] /sql on z-dat-s6-a is UNKNOWN: CHECK_NRPE: Received 0 bytes from daemon. Check the remote server logs for error messages. [03:54:44] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 70228 MB (7% inode=98%): [03:54:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 70227 MB (7% inode=98%): [04:04:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [04:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:24:43] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [04:28:44] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [04:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [04:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [04:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [04:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [04:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [05:04:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:24:15] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [05:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [05:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [05:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [05:40:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [05:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [05:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [05:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [06:04:24] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [06:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:24:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:25:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [06:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [06:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [06:40:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [06:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [06:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [06:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [07:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [07:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [07:28:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [07:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [07:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [07:41:15] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [07:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [07:43:23] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:04:33] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:13:13] s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2008.000000 [08:13:23] MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2023 [08:24:13] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [08:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:27:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [08:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [08:32:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35385 MB (8% inode=99%): [08:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [08:40:55] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [08:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [08:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [08:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [08:46:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3646.000000 [08:46:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3660 [09:04:33] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:18:14] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 99355 MB (10% inode=99%): [09:24:13] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [09:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:25:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [09:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [09:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [09:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [09:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [09:42:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [09:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [09:46:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7230.000000 [09:46:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7244 [10:04:33] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:24:13] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [10:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [10:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [10:29:54] RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [10:34:23] RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0 [10:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [10:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [10:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [10:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [10:46:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 10792.000000 [10:46:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10806 [11:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [11:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:24:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:27:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [11:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [11:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [11:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [11:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [11:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [11:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (1 warning): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [11:44:54] @replag [11:44:54] Betacommand: s1-rr-a: 46s [-0.00 s/s]; s1-user: 46s [-0.00 s/s] [11:46:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 14340.000000 [11:46:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14353 [11:49:11] @replag [11:49:12] Betacommand: s2-user: 11s [-0.00 s/s] [11:55:04] John * [Toolserver-l] Query killer and replag [12:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:24:13] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [12:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:25:04] DaB. * Re: [Toolserver-l] Query killer and replag [12:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:27:04] John * Re: [Toolserver-l] Query killer and replag [12:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [12:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [12:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [12:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [12:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [12:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [12:43:23] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [12:46:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17840.000000 [12:46:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17853 [12:53:04] Platonides * Re: [Toolserver-l] Query killer and replag [12:56:04] John * Re: [Toolserver-l] Query killer and replag [13:01:24] Betacommand you have a minute? [13:04:33] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:05:26] or any toolserver contributor for that matter [13:06:41] * Platonides is here, ToAruShiroiNeko [13:06:55] Platonides, ah great! [13:07:26] I am looking for a tool or query that would be fed 44 accounts and it would display a graph of their first and last edit. [13:07:29] can this be done? [13:08:19] a bit like https://en.wikipedia.org/w/index.php?title=User:White_Cat/RFAR/graph probably less detailed [13:08:31] just the dates? [13:09:00] dates would help but such a graph would help more to establish which account was used for how long and how many at the same time [13:09:13] there are 44 accounts which is why this is confusing [13:09:58] http://pastebin.com/eYX994aH [13:10:49] I'm a bit uncomfortable of making a general purpose tool, see the rule "Tools that allow profiling of individual user's activity (beyond what can easily be achieved directly on the public wiki sites) must only be applied with the respective user's consent (opt-in)." [13:10:51] arbcom limited him to one account [13:11:16] enwiki, I guess? [13:11:18] all of this is one users account [13:11:20] yes enwiki [13:11:32] these are what he posts as his sockpuppets [13:12:00] he even posted his passwords a while back ago just to get them compromised [13:12:22] the only information tool would show is first and last edit date per account [13:12:43] and weather or not they are locked/blocked after last edit maybe unless this is too hard [13:13:28] I dont think this is profiling. [13:17:00] I'm on it [13:17:06] yay :) [13:17:17] it might be best to have multiple graphs like I had [13:17:26] he has 44 accounts distributed from 2005-2012 [13:17:37] so squishing it in one thing may be unreadable :/ [13:18:01] but thats your call [13:21:49] this relates to the discussion https://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard#Proposed_community_ban_for_Jack_Merridew.2FDavenbelle.2FBr.27er_Rabbit.2F... [13:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [13:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:24:27] Garrafa-azul and Ogoh-ogoh don't have contributions [13:24:31] (probably deleted) [13:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:25:26] possibly [13:25:43] account creation date and block date can still apply :/ [13:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:26:08] he seems to create sleeper sockpuppet accounts to from time to time [13:26:18] raw values at http://toolserver.org/~platonides/accts44/accts44.php [13:26:35] now, you wanted it sorted by beginning date? [13:26:51] yeah that would be great [13:27:33] consider my graph link. I have accounts listed in chronologic order [13:27:44] its a mere wikitable [13:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [13:28:10] if you could denote blocked/locked dates somehow it would also be great [13:28:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [13:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [13:40:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [13:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [13:42:51] Garrafa-azul and Ogoh-ogoh don't have delted contributions either [13:42:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [13:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [13:44:13] but they have a date of creation and block right? [13:44:44] those two are probably not important on their own but it would put their date of creation and block into perspective [13:46:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21129.000000 [13:46:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21141 [13:46:44] I added them [13:46:49] yay [13:46:59] the page is very slow to load, though [13:47:08] thats ok its a lot of accounts [13:47:36] and not specially optimised :P [13:51:03] woha [13:51:08] it is quite slow :D [13:51:38] didnt load anything, not sure if it is complete [14:00:33] it shows the four fields now: http://toolserver.org/~platonides/accts44/accts44.php [14:03:33] looking [14:04:33] it should pick registration date or first edit as the opening date (whicever is first) [14:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:06:28] I am showing both [14:06:34] but sorting by first edit [14:06:49] note that in some cases they were sleeping for a long time: 2007-12-21 12:53:36 2010-06-10 19:40:53 [14:07:56] hmm [14:08:07] I dont know if you can denote such big gaps [14:09:24] https://en.wikipedia.org/w/index.php?title=Special:Log/block&page=User%3ABarong [14:09:29] this account is locked [14:09:36] even though it doesnt appear blocked [14:09:47] its globally locked thouhg [14:10:39] I was only looking at local blocks [14:11:17] ah [14:11:37] he had accounts where he entirely stopped using for years [14:11:52] we didnt have any proof that they were sockpuppets [14:11:58] he claimed he lost the passwords [14:12:17] blocks were symbolic in such cases [14:13:24] in some cases he was locked before the block [14:13:34] though that applies to his accounts after 2011 probably [14:16:09] to put in a few words, his behaviour is immensely complex to follow [14:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [14:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:25:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [14:28:43] I am now also listing when the account is globally locked [14:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [14:29:31] you are awesome :) [14:30:12] thanks [14:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [14:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [14:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [14:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [14:43:23] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [14:46:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 23800.000000 [14:46:23] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23812 [15:00:03] I have updated the graph to give you an idea of UI :) [15:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:15:14] @replag [15:15:15] Dispenser: No replag found. Use "@replag all" to see all lags. [15:15:38] anyone available who's able to add some jars to tomcat? [15:16:08] Platonides do you see what change I made? [15:16:16] https://jira.toolserver.org/browse/TS-1550 [15:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [15:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:25:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [15:28:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [15:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [15:34:19] Platonides they dont seem to be order properly at times :/ [15:37:56] Platonides actually could you make them sortable? [15:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [15:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [15:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [15:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [15:46:19] @replag all [15:46:19] Dispenser: s1-rr-a: 55s [+0.00 s/s]; s1-user: 55s [+0.00 s/s]; s2-user: 10s [-0.00 s/s]; s2-user-c: 1s [-0.00 s/s]; s3-rr-a: 15s [-0.00 s/s]; s3-user: 15s [-0.00 s/s]; s4-rr-a: 1s [-0.00 s/s]; s4-user: 1s [-0.00 s/s] [15:46:21] Dispenser: s5-rr-a: 1s [-0.00 s/s]; s5-user: 1s [-0.00 s/s]; s5-user-c: 1s [-0.00 s/s]; s6-rr-a: 1s [-0.00 s/s]; s6-user: 1s [-0.00 s/s]; s7-rr-a: 0s [-0.00 s/s]; s7-user: 0s [-0.00 s/s] [15:46:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27179.000000 [15:46:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27191 [15:47:35] replag for sql-s1-rr (thyme) is wrong, its nearly 7.5 hours [15:54:44] /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 69033 MB (7% inode=98%): [15:54:54] /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 69033 MB (7% inode=98%): [16:01:15] @replag [16:01:15] liangent: s3-rr-a: 11s [-0.00 s/s]; s3-user: 11s [-0.00 s/s] [16:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [16:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:25:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [16:28:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [16:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [16:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [16:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [16:42:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [16:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [16:46:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30620 [16:47:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 30666.000000 [17:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [17:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:24:53] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [17:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [17:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [17:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [17:40:53] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [17:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [17:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [17:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [17:46:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34159 [17:47:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34205.000000 [18:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [18:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:27:53] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [18:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [18:33:33] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [18:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [18:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [18:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [18:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [18:46:24] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 37733 [18:47:23] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 37779.000000 [19:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:09:13] hello all [19:17:35] hello, by me, the http://bots.wmflabs.org/~richs/commonshelper.php are not go. (Down) Is that right? [19:18:29] You probably want #wikimedia-labs ? [19:18:38] heubergen: ^ [19:22:00] Platonides you there? :/ [19:22:22] define down, it seems up to me [19:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [19:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [19:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [19:30:39] DaBPunkt: can you take a look at the replag issues on s1 ? [19:30:47] @replag [19:30:47] Betacommand: No replag found. Use "@replag all" to see all lags. [19:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [19:35:22] @replag all [19:35:22] ToAruShiroiNeko: s1-rr-a: 0s [-0.00 s/s]; s1-user: 0s [-0.00 s/s]; s2-user: 0s [-0.00 s/s]; s2-user-c: 2s [+0.00 s/s]; s3-rr-a: 2s [-0.00 s/s]; s3-user: 2s [-0.00 s/s]; s4-rr-a: 2s [+0.00 s/s]; s4-user: 2s [+0.00 s/s] [19:35:23] ToAruShiroiNeko: s5-rr-a: 1s [-]; s5-user: 1s [-]; s5-user-c: 2s [+0.00 s/s]; s6-rr-a: 1s [-]; s6-user: 1s [-]; s7-rr-a: 0s [-]; s7-user: 0s [-] [19:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [19:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [19:41:17] Betacommand: let me look [19:42:18] it swaps [19:42:28] I will restart mysql there [19:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [19:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [19:46:24] MySQL slave on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [19:48:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on thyme (146) [19:50:24] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [19:57:34] is there anything going wrong with user account creations at the toolserver Wiki? last 500 wiki changes seems to be account creations, looks a little bit suspicious [19:57:41] https://wiki.toolserver.org/w/index.php?title=Special:RecentChanges&limit=500 [19:58:18] probably bots [19:59:01] but 500 in October alone? well... okay... ;) [20:00:19] 57 today... [20:00:26] that's a bit too much [20:00:41] but it has just a math captcha [20:01:23] MySQL on thyme is OK: Uptime: 23 Threads: 5 Questions: 76 Slow queries: 0 Opens: 44 Flush tables: 1 Open tables: 37 Queries per second avg: 3.304 [20:04:33] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:16:02] Platonides: Ive even seen reCaptcha bypassed [20:19:12] @replag [20:19:12] Betacommand: s1-rr-a: 15s [+0.01 s/s]; s1-user: 15s [+0.01 s/s]; s3-rr-a: 22s [+0.01 s/s]; s3-user: 22s [+0.01 s/s] [20:20:49] I have heard of such reports, too [20:21:14] there is a wiki I work with that is getting hammered [20:21:33] @replag [20:21:36] DaBPunkt: s1-rr-a: 11h 5m 51s [-] [20:21:48] now it tells the truth [20:21:57] DaBPunkt: why was tsbot's report so off? [20:22:14] Betacommand: used rosemary's replag instead of thymes [20:23:42] Betacommand: used rosemary's replag instead of thymes [20:23:45] sorry [20:23:47] @replag [20:23:47] DaBPunkt: s1-rr-a: 10h 59m 13s [-3.00 s/s]; s1-user: 15s [-]; s3-rr-a: 51s [-]; s3-user: 51s [-] [20:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [20:24:23] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:25:18] Betacommand, have you tried fancycaptcha ? [20:25:31] Platonides: no [20:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:25:46] you can give it a try [20:26:12] also, questycaptcha is very strong [20:26:22] you manually specify the question [20:26:31] "What's the company running this wiki?" [20:26:43] It's unclear how much it inhibits legitimate users, though. [20:27:02] http://captcha.civilrightsdefenders.org/ [20:27:10] Another approach. :-) ^ [20:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [20:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [20:31:42] «The parliament in St. Petersburg recently passed a law that forbids "homosexual propaganda". How does that make you feel?» [20:31:56] that's highly dependant on whethere I'm gay or homophobic [20:32:11] (actually, I don't know what it considers homosexual propaganda...) [20:32:34] /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 35531 MB (8% inode=99%): [20:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [20:36:03] it's an interesting concept, though [20:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [20:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [20:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [20:43:23] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [20:46:34] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 36066 [20:48:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 35990.000000 [21:04:33] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:18:13] /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 98221 MB (10% inode=99%): [21:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [21:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:25:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:25:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [21:28:53] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [21:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [21:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [21:41:13] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [21:42:53] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [21:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [21:46:33] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 30015 [21:48:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 29813.000000 [22:04:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [22:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:25:13] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [22:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [22:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [22:40:54] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [22:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [22:42:54] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [22:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [22:46:34] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21415 [22:48:13] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 21144.000000 [23:04:33] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:08:29] nacht ts [23:24:14] SRaid on nightshade is CRITICAL: NRPE: Unable to read output [23:24:24] s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:24:54] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:25:14] MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:25:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:27:54] FMA on amaranth is CRITICAL: Failed components: hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7 [23:28:54] SMF on web.amaranth is CRITICAL: ERROR - maintenance: svc:/application/jira:default [23:33:34] DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs [23:41:03] Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output [23:41:14] Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output [23:43:03] Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output [23:43:24] CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 34:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42: [23:46:34] MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12050 [23:48:14] s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 11801.000000