[00:01:01] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[00:12:41] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44665.000000  
[00:21:50] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 63607.000000  
[00:22:11] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 63465  
[00:23:30] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74192 MB (12% inode=99%):  
[00:23:40] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[00:23:51] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[00:24:11] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[00:24:21] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[00:24:21] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[00:24:30] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[00:24:31] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[00:24:40] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[00:24:50] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[00:24:50] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[00:25:20] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[00:27:21] <tsnag>	 /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37133 MB (9% inode=99%):  
[00:27:31] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73720 MB (7% inode=98%):  
[00:27:31] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73720 MB (7% inode=98%):  
[00:57:17] <DaBPunkt>	 nacht ts
[01:01:00] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[01:12:40] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 44351.000000  
[01:21:51] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 46544.000000  
[01:22:10] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 46406  
[01:23:31] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74127 MB (12% inode=99%):  
[01:23:41] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[01:23:51] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[01:24:10] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[01:24:20] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[01:24:21] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[01:24:30] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[01:24:30] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[01:24:40] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[01:24:50] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[01:24:51] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[01:25:21] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[01:54:01] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[01:54:01] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[01:54:50] <tsnag>	 /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[01:55:21] <tsnag>	 /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 36417 MB (8% inode=99%):  
[01:55:31] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73570 MB (7% inode=98%):  
[01:55:31] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73570 MB (7% inode=98%):  
[02:01:01] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[02:09:51] <tsnag>	 RAID on adenia is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[02:12:41] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 43400.000000  
[02:21:51] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 34791.000000  
[02:22:10] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 34723  
[02:23:31] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 74060 MB (12% inode=99%):  
[02:23:41] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[02:23:50] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[02:24:10] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[02:24:20] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[02:24:20] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[02:24:30] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[02:24:30] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[02:24:40] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[02:24:50] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[02:24:51] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[02:25:21] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[02:29:21] <tsnag>	 RAID on adenia is OK: OK - TOTAL: 2: FAILED: 0: DEGRADED: 0  
[03:01:01] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[03:12:41] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 38906.000000  
[03:21:54] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 31075.000000  
[03:22:23] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 31093  
[03:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73959 MB (12% inode=99%):  
[03:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[03:23:53] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[03:24:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[03:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[03:24:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[03:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[03:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[03:24:44] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[03:24:54] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[03:24:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[03:25:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[04:01:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[04:08:35] <Matthew_>	 @replag s1
[04:08:35] <tsbot>	 Matthew_: s2-user: 2d 5h 11m 12s [-0.05 s/s]; s2-user-c: 8h 9m 27s [-1.63 s/s]; s3-rr-a: 47s [-0.00 s/s]; s3-user: 47s [-0.00 s/s]; s5-user-c: 8h 9m 27s [-1.63 s/s]; s7-rr-a: 11s [-0.00 s/s]; s7-user: 11s [-0.00 s/s]
[04:08:43] <Matthew_>	 @replag all
[04:08:44] <tsbot>	 Matthew_: s1-rr-a: 3s [-0.00 s/s]; s1-user: 3s [-0.00 s/s]; s2-user: 2d 5h 10m 35s [-4.61 s/s]; s2-user-c: 8h 9m 12s [-1.86 s/s]; s3-rr-a: 55s [+0.99 s/s]; s3-user: 55s [+0.99 s/s]; s4-rr-a: 8s [+0.00 s/s]; s4-user: 8s [+0.00 s/s]
[04:08:44] <tsbot>	 Matthew_: s5-rr-a: 1s [-0.00 s/s]; s5-user: 1s [-0.00 s/s]; s5-user-c: 8h 9m 12s [-1.86 s/s]; s6-rr-a: 12s [-0.00 s/s]; s6-user: 12s [-0.00 s/s]; s7-rr-a: 7s [-0.50 s/s]; s7-user: 7s [-0.50 s/s]
[04:12:44] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 28765.000000  
[04:14:30] <Matthew_>	 OK, what's up with Thyme? It appears to have a huge replag.   But it's not showing here... 
[04:21:54] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 33783.000000  
[04:22:23] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 33810  
[04:22:36] <Matthew_>	 Wait.... how come that's not showing on replag then?
[04:23:01] <Matthew_>	 @replag all
[04:23:02] <tsbot>	 Matthew_: s1-rr-a: 1s [-0.00 s/s]; s1-user: 1s [-0.00 s/s]; s2-user: 2d 4h 49m 8s [-1.50 s/s]; s2-user-c: 7h 29m 19s [-2.79 s/s]; s3-rr-a: 8m 30s [+0.53 s/s]; s3-user: 8m 30s [+0.53 s/s]; s4-rr-a: 2s [-0.01 s/s]; s4-user: 2s [-0.01 s/s]
[04:23:03] <tsbot>	 Matthew_: s5-rr-a: 4s [+0.00 s/s]; s5-user: 4s [+0.00 s/s]; s5-user-c: 7h 29m 19s [-2.79 s/s]; s6-rr-a: 2s [-0.01 s/s]; s6-user: 2s [-0.01 s/s]; s7-rr-a: 1s [-0.01 s/s]; s7-user: 1s [-0.01 s/s]
[04:23:27] <legoktm>	 Hm
[04:23:34] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73859 MB (12% inode=99%):  
[04:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[04:23:53] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[04:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[04:24:24] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[04:24:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[04:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[04:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[04:24:44] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[04:24:54] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[04:24:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[04:25:23] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[05:01:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[05:05:44] <Matthew_>	 @replag all
[05:05:45] <tsbot>	 Matthew_: s1-rr-a: 8m 9s [+0.19 s/s]; s1-user: 8m 9s [+0.19 s/s]; s2-user: 2d 3h 19m 2s [-2.11 s/s]; s2-user-c: 4h 17m 34s [-4.49 s/s]; s3-rr-a: 2m 26s [-0.14 s/s]; s3-user: 2m 26s [-0.14 s/s]; s4-rr-a: 9s [+0.00 s/s]; s4-user: 9s [+0.00 s/s]
[05:05:46] <tsbot>	 Matthew_: s5-rr-a: 2s [-0.00 s/s]; s5-user: 3s [-0.00 s/s]; s5-user-c: 4h 17m 33s [-4.49 s/s]; s6-rr-a: 1s [-0.00 s/s]; s6-user: 1s [-0.00 s/s]; s7-rr-a: 19s [+0.01 s/s]; s7-user: 19s [+0.01 s/s]
[05:12:44] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 13864.000000  
[05:21:53] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 27910.000000  
[05:22:24] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 27814  
[05:23:34] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73785 MB (12% inode=99%):  
[05:23:44] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[05:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[05:24:25] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[05:24:25] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[05:24:25] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[05:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[05:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[05:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[05:24:54] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[05:24:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[05:25:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[06:01:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[06:03:33] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2175  
[06:03:34] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2181.000000  
[06:12:43] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5778.000000  
[06:21:54] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 17848.000000  
[06:22:24] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 17830  
[06:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73713 MB (12% inode=99%):  
[06:23:44] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[06:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[06:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[06:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[06:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[06:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[06:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[06:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[06:25:24] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[06:25:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[06:25:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[06:55:44] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3398.000000  
[07:01:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[07:03:33] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2422  
[07:03:33] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2427.000000  
[07:17:24] <tsnag>	 MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3597  
[07:17:53] <tsnag>	 s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3376.000000  
[07:19:33] <tsnag>	 MySQL slave on rosemary is OK: Uptime: 15436928  Threads: 11  Questions: 6841404815  Slow queries: 2224705  Opens: 317489  Flush tables: 6  Open tables: 4169  Queries per second avg: 443.184 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1800  
[07:19:33] <tsnag>	 s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1781.000000  
[07:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73621 MB (12% inode=99%):  
[07:23:44] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[07:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[07:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[07:24:24] <tsnag>	 MySQL slave on thyme is OK: Uptime: 9285805  Threads: 11  Questions: 3070952224  Slow queries: 1044369  Opens: 294132  Flush tables: 2  Open tables: 643  Queries per second avg: 330.714 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1625  
[07:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[07:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[07:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[07:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[07:24:53] <tsnag>	 s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1421.000000  
[07:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[07:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[07:25:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[07:25:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[07:48:52] <legoktm>	 !replag
[07:49:00] <legoktm>	 ugh
[07:49:03] <legoktm>	 @replag
[07:49:04] <tsbot>	 legoktm: s1-rr-a: 29s [-1.44 s/s]; s1-user: 29s [-1.44 s/s]; s2-user: 2d 3h 24m 24s [+0.26 s/s]; s2-user-c: 44m 42s [-0.21 s/s]; s5-user-c: 44m 42s [-0.21 s/s]
[07:55:44] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2757.000000  
[08:01:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[08:23:34] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73472 MB (12% inode=99%):  
[08:23:44] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[08:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[08:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[08:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[08:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[08:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[08:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[08:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[08:25:03] <tsnag>	 CAM on hemlock is CRITICAL: Connection refused by host  
[08:25:24] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[08:25:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[08:25:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[08:37:03] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[08:38:00] <akoopal>	 @replag
[08:38:06] <tsbot>	 akoopal: s1-rr-a: 4m 5s [+0.07 s/s]; s1-user: 4m 5s [+0.07 s/s]; s2-user: 2d 4h 20s [+0.73 s/s]; s2-user-c: 43m 53s [-0.02 s/s]; s3-rr-a: 25s [+0.00 s/s]; s3-user: 25s [+0.00 s/s]; s5-user-c: 43m 54s [-0.02 s/s]
[08:45:05] <reba>	 Tim Landscheidt * Re: [Toolserver-l] How to have qsub mail output?
[08:55:43] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2406.000000  
[09:00:54] <tsnag>	 Load avg. on adenia is WARNING: WARNING - load average: 17.31, 14.49, 9.36  
[09:06:53] <tsnag>	 s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2155.000000  
[09:07:24] <tsnag>	 MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2183  
[09:17:54] <tsnag>	 /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 96021 MB (9% inode=99%):  
[09:23:34] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73340 MB (12% inode=99%):  
[09:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[09:23:53] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[09:24:23] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[09:24:33] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[09:24:33] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[09:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[09:24:54] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[09:24:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[09:25:24] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[09:25:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[09:25:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[09:31:23] <tsnag>	 MySQL slave on thyme is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3607  
[09:31:54] <tsnag>	 s1 replag on thyme is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3637.000000  
[09:37:03] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[09:47:05] <reba>	 Marlen Caemmerer * [Toolserver-announce] Maintenance sql-user / host adenia 10th October
[09:50:53] <tsnag>	 Load avg. on adenia is OK: OK - load average: 12.24, 14.39, 14.98  
[09:51:14] <reba>	 Marlen Caemmerer * [Toolserver-announce] Maintenance sql-s1-user /rosemary 12th October
[09:52:43] <tsnag>	 SSH on hemlock is CRITICAL: Server answer:  
[09:54:54] <tsnag>	 /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[09:55:04] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[09:55:04] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[09:55:24] <tsnag>	 /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37624 MB (9% inode=99%):  
[09:55:33] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73715 MB (7% inode=98%):  
[09:55:33] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73715 MB (7% inode=98%):  
[09:55:43] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2902.000000  
[09:57:03] <tsnag>	 CAM on hemlock is CRITICAL: Connection refused by host  
[09:57:24] <tsnag>	 NTP on hemlock is CRITICAL: NTP CRITICAL: No response from NTP server  
[10:01:03] <tsnag>	 Environment IPMI on hemlock is CRITICAL: Connection refused by host  
[10:01:54] <tsnag>	 s1 replag on thyme is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3598.000000  
[10:02:24] <tsnag>	 MySQL slave on thyme is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3449  
[10:02:24] <tsnag>	 Load avg. on hemlock is CRITICAL: Connection refused by host  
[10:02:33] <tsnag>	 / on hemlock is CRITICAL: Connection refused by host  
[10:02:43] <tsnag>	 /aux0 on hemlock is CRITICAL: Connection refused by host  
[10:02:53] <tsnag>	 /home on hemlock is CRITICAL: Connection refused by host  
[10:02:53] <tsnag>	 /tmp on hemlock is CRITICAL: Connection refused by host  
[10:08:44] <tsnag>	 SSH on hemlock is OK: SSH OK - OpenSSH_5.8p2-hpn13v11 (protocol 2.0)  
[10:12:54] <tsnag>	 s1 replag on thyme is OK: QUERY OK: SELECT ts_rc_age() returned 1703.000000  
[10:13:23] <tsnag>	 MySQL slave on thyme is OK: Uptime: 9295945  Threads: 4  Questions: 3072165645  Slow queries: 1046397  Opens: 294159  Flush tables: 2  Open tables: 650  Queries per second avg: 330.484 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1576  
[10:20:43] <tsnag>	 SMTP on hemlock is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[10:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73195 MB (12% inode=99%):  
[10:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[10:23:53] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[10:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[10:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[10:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[10:24:44] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[10:24:54] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[10:24:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[10:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:25:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[10:25:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[10:53:09] <DaBPunkt>	 hello all
[10:53:53] <tsnag>	 /sql on z-dat-s7-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:54:03] <tsnag>	 /sql on z-dat-s6-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:54:04] <tsnag>	 /sql on z-dat-s3-a is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[10:55:43] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2939.000000  
[10:56:06] <DaBPunkt>	 @replag
[10:56:08] <tsbot>	 DaBPunkt: s1-rr-a: 12m 7s [+0.16 s/s]; s1-user: 12m 7s [+0.16 s/s]; s2-user: 2d 5h 22m 25s [+0.56 s/s]; s2-user-c: 49m 6s [-0.03 s/s]; s3-rr-a: 3m 37s [+0.02 s/s]; s3-user: error; s5-user-c: 49m 7s [-0.03 s/s]; s6-rr-a: 3m 36s [+0.01 s/s]
[10:56:10] <tsbot>	 DaBPunkt: s6-user: error; s7-rr-a: 3m 16s [+0.01 s/s]; s7-user: 3m 16s [+0.01 s/s]
[10:56:24] <tsnag>	 /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 37534 MB (9% inode=99%):  
[10:56:34] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 73553 MB (7% inode=98%):  
[10:56:34] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 73553 MB (7% inode=98%):  
[10:57:03] <tsnag>	 CAM on hemlock is CRITICAL: Connection refused by host  
[10:57:23] <tsnag>	 NTP on hemlock is CRITICAL: NTP CRITICAL: No response from NTP server  
[11:13:44] <tsnag>	 /aux0 on hemlock is CRITICAL: Connection refused by host  
[11:13:54] <tsnag>	 /home on hemlock is CRITICAL: Connection refused by host  
[11:13:54] <tsnag>	 /tmp on hemlock is CRITICAL: Connection refused by host  
[11:14:04] <tsnag>	 Environment IPMI on hemlock is CRITICAL: Connection refused by host  
[11:14:24] <tsnag>	 Load avg. on hemlock is CRITICAL: Connection refused by host  
[11:14:33] <tsnag>	 / on hemlock is CRITICAL: Connection refused by host  
[11:21:34] <tsnag>	 SMTP on hemlock is CRITICAL: Connection refused  
[11:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 73046 MB (11% inode=99%):  
[11:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[11:23:53] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[11:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[11:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[11:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[11:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[11:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[11:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[11:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[11:25:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[11:25:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[11:37:34] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2018  
[11:37:43] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2020.000000  
[11:37:43] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3626.000000  
[11:42:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[11:42:04] <tsnag>	 Environment IPMI on hemlock is OK: ok:  temperature ok fan ok voltage ok chassis ok  
[11:42:24] <tsnag>	 Load avg. on hemlock is OK: OK - load average: 0.47, 0.52, 0.41  
[11:42:34] <tsnag>	 / on hemlock is OK: DISK OK - free space: / 6931 MB (34% inode=90%):  
[11:42:34] <tsnag>	 SMTP on hemlock is OK: SMTP OK - 0.084 sec. response time  
[11:42:43] <tsnag>	 /aux0 on hemlock is OK: DISK OK - free space: / 6910 MB (34% inode=90%):  
[11:42:53] <tsnag>	 /home on hemlock is OK: DISK OK - free space: /home 18871 MB (37% inode=87%):  
[11:42:53] <tsnag>	 /tmp on hemlock is OK: DISK OK - free space: / 6907 MB (34% inode=90%):  
[11:44:23] <tsnag>	 NTP on hemlock is OK: NTP OK: Offset 0.006424 secs  
[12:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72894 MB (11% inode=99%):  
[12:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[12:23:53] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[12:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[12:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[12:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[12:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[12:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[12:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[12:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[12:25:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[12:25:33] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[12:25:43] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3570.000000  
[12:37:34] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1814  
[12:37:43] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1814.000000  
[12:42:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[12:42:43] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3632.000000  
[12:43:34] <tsnag>	 MySQL slave on rosemary is OK: Uptime: 15456367  Threads: 20  Questions: 6848224276  Slow queries: 2232214  Opens: 318472  Flush tables: 6  Open tables: 4150  Queries per second avg: 443.68 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1800  
[12:48:44] <tsnag>	 s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1794.000000  
[13:01:34] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2048  
[13:01:44] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2051.000000  
[13:20:35] <tsnag>	 MySQL slave on rosemary is OK: Uptime: 15458587  Threads: 10  Questions: 6849003041  Slow queries: 2232957  Opens: 318625  Flush tables: 6  Open tables: 4149  Queries per second avg: 443.54 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1760  
[13:20:44] <tsnag>	 s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1759.000000  
[13:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72741 MB (11% inode=99%):  
[13:23:42] <DaBPunkt>	 @replag
[13:23:43] <tsbot>	 DaBPunkt: s1-rr-a: 27m 39s [-0.67 s/s]; s1-user: 27m 39s [-0.67 s/s]; s2-user: 2d 6h 23m 15s [+0.62 s/s]; s2-user-c: 1h 30m 46s [+0.84 s/s]; s3-rr-a: 12s [+0.01 s/s]; s3-user: 12s [+0.01 s/s]; s5-user-c: 1h 30m 46s [+0.84 s/s]
[13:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[13:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[13:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[13:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[13:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[13:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[13:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[13:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[13:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[13:26:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[13:26:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[13:31:53] <tsnag>	 Load avg. on adenia is WARNING: WARNING - load average: 18.04, 15.04, 10.42  
[13:38:53] <tsnag>	 Load avg. on adenia is OK: OK - load average: 7.88, 13.39, 11.72  
[13:42:03] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[13:42:43] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 5980.000000  
[14:01:34] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1846  
[14:01:43] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 1845.000000  
[14:01:43] <tsnag>	 s4 replag on cassia is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3485.000000  
[14:03:43] <tsnag>	 s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1797.000000  
[14:04:34] <tsnag>	 MySQL slave on rosemary is OK: Uptime: 15461228  Threads: 21  Questions: 6849802053  Slow queries: 2234066  Opens: 318896  Flush tables: 6  Open tables: 4158  Queries per second avg: 443.30 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1778  
[14:10:43] <tsnag>	 s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 1697.000000  
[14:14:30] <DaBPunkt>	 @replag
[14:14:31] <tsbot>	 DaBPunkt: s1-rr-a: 24m 43s [-0.06 s/s]; s1-user: 24m 43s [-0.06 s/s]; s2-user: 2d 5h 50m 18s [-0.65 s/s]; s2-user-c: 20m 37s [-1.38 s/s]; s3-rr-a: 12s [-]; s3-user: 12s [-]; s5-user-c: 20m 37s [-1.38 s/s]
[14:23:34] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72583 MB (11% inode=99%):  
[14:23:44] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[14:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[14:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[14:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[14:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[14:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[14:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[14:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[14:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[14:26:23] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[14:26:34] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[14:29:43] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2144.000000  
[14:30:33] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2195  
[14:42:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[14:51:33] <DaBPunkt>	 @replag
[14:51:35] <tsbot>	 DaBPunkt: s1-rr-a: 56m 41s [+0.97 s/s]; s1-user: 56m 42s [+0.97 s/s]; s2-user: 2d 5h 28m 38s [-1.40 s/s]; s3-rr-a: 31s [+0.02 s/s]; s3-user: 31s [+0.02 s/s]
[14:55:34] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3628  
[14:55:44] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3635.000000  
[14:55:54] <tsnag>	 s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 2146.000000  
[14:58:37] <DaBPunkt>	 @replag
[14:58:38] <tsbot>	 DaBPunkt: s1-rr-a: 1h 3m 33s [+0.98 s/s]; s1-user: 1h 3m 33s [+0.99 s/s]; s2-user: 2d 5h 30m 1s [+0.33 s/s]; s3-rr-a: 1m 33s [+0.99 s/s]; s3-user: 1m 33s [+0.99 s/s]
[15:21:54] <tsnag>	 s4 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 3649.000000  
[15:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72421 MB (11% inode=99%):  
[15:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[15:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[15:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[15:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[15:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[15:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[15:24:50] <DaBPunkt>	 looks like cassia died. I will look
[15:24:54] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[15:24:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[15:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[15:26:23] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[15:26:33] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[15:28:49] <DaBPunkt>	 looks like it had just a hickup
[15:29:44] <tsnag>	 s5 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146)  
[15:29:44] <tsnag>	 s4 replag on cassia is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on cassia (146)  
[15:30:23] <tsnag>	 MySQL slave on cassia is CRITICAL: Cant connect to MySQL server on cassia (146)  
[15:30:23] <tsnag>	 MySQL on cassia is CRITICAL: Cant connect to MySQL server on cassia (146)  
[15:36:39] <DaBPunkt>	 cassia's mysql crashed, doesn self-repair at the moment
[15:36:44] <DaBPunkt>	 -n
[15:37:44] <tsnag>	 s5 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 836.000000  
[15:37:44] <tsnag>	 s4 replag on cassia is OK: QUERY OK: SELECT ts_rc_age() returned 845.000000  
[15:38:24] <tsnag>	 MySQL slave on cassia is OK: Uptime: 680  Threads: 24  Questions: 7857  Slow queries: 2  Opens: 282  Flush tables: 1  Open tables: 268  Queries per second avg: 11.554 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 838  
[15:38:24] <tsnag>	 MySQL on cassia is OK: Uptime: 680  Threads: 23  Questions: 7859  Slow queries: 2  Opens: 282  Flush tables: 1  Open tables: 268  Queries per second avg: 11.557  
[15:42:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[15:45:53] <tsnag>	 Load avg. on adenia is WARNING: WARNING - load average: 18.18, 15.95, 11.88  
[15:50:53] <tsnag>	 s4 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3544.000000  
[15:55:34] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 6622  
[15:55:44] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 6623.000000  
[15:55:53] <tsnag>	 Load avg. on adenia is OK: OK - load average: 8.02, 13.68, 13.45  
[16:01:53] <tsnag>	 s4 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1726.000000  
[16:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72253 MB (11% inode=99%):  
[16:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[16:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[16:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[16:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[16:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[16:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[16:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[16:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[16:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[16:26:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[16:26:33] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[16:42:03] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[16:55:34] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8024  
[16:55:44] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8026.000000  
[17:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 72032 MB (11% inode=99%):  
[17:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[17:23:53] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[17:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[17:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[17:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[17:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[17:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[17:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[17:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[17:26:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[17:27:23] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[17:42:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[17:55:33] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9183  
[17:55:44] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9184.000000  
[18:23:34] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71751 MB (11% inode=99%):  
[18:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[18:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[18:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[18:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[18:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[18:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[18:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[18:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[18:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[18:26:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[18:27:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[18:42:03] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[18:55:33] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9088  
[18:55:44] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 9090.000000  
[19:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71527 MB (11% inode=99%):  
[19:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[19:23:53] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[19:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[19:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[19:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[19:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[19:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[19:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[19:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[19:27:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[19:32:23] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[19:42:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[19:55:34] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8786  
[19:55:43] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8791.000000  
[20:23:34] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 71273 MB (11% inode=99%):  
[20:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[20:23:53] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[20:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[20:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[20:24:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[20:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[20:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[20:24:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[20:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[20:27:33] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[20:32:23] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[20:42:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[20:55:33] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8688  
[20:55:43] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 8686.000000  
[21:17:53] <tsnag>	 /sql on rosemary is CRITICAL: DISK CRITICAL - free space: /sql 103519 MB (10% inode=99%):  
[21:23:33] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70937 MB (11% inode=99%):  
[21:23:43] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[21:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[21:24:24] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[21:24:34] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[21:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[21:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[21:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[21:25:23] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[21:25:33] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[21:28:24] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[21:32:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[21:42:04] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[21:55:33] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7196  
[21:55:44] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 7202.000000  
[22:23:34] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70673 MB (11% inode=99%):  
[22:23:44] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[22:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[22:24:23] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[22:24:33] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[22:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[22:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[22:24:53] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[22:25:24] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[22:25:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[22:28:33] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[22:32:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[22:40:15] <Merlissimo>	 @replag
[22:40:16] <tsbot>	 Merlissimo: s1-rr-a: 1h 22m 34s [-0.29 s/s]; s1-user: 1h 22m 34s [-0.29 s/s]; s2-user: 1d 14h 50m 33s [-2.64 s/s]; s3-rr-a: 43s [+0.00 s/s]; s3-user: 44s [+0.00 s/s]
[22:42:03] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[22:55:33] <tsnag>	 MySQL slave on rosemary is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4789  
[22:55:43] <tsnag>	 s1 replag on rosemary is CRITICAL: QUERY CRITICAL: SELECT ts_rc_age() returned 4793.000000  
[22:56:23] <tsnag>	 /sql on z-dat-s7-a is WARNING: DISK WARNING - free space: /sql 36667 MB (9% inode=99%):  
[22:56:34] <tsnag>	 /sql on z-dat-s6-a is WARNING: DISK WARNING - free space: /sql 72918 MB (7% inode=98%):  
[22:56:34] <tsnag>	 /sql on z-dat-s3-a is WARNING: DISK WARNING - free space: /sql 72918 MB (7% inode=98%):  
[23:10:33] <tsnag>	 MySQL slave on rosemary is WARNING: SLOW_SLAVE WARNING: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3515  
[23:10:43] <tsnag>	 s1 replag on rosemary is WARNING: QUERY WARNING: SELECT ts_rc_age() returned 3487.000000  
[23:23:34] <tsnag>	 /sql on ptolemy is WARNING: DISK WARNING - free space: /sql 70478 MB (11% inode=99%):  
[23:23:44] <tsnag>	 FMA on amaranth is CRITICAL: Failed components:  hc://:product-id=SUN-FIRE-X4150:server-id=amaranth:chassis-id=0819QAR1D1:serial=518545072303039020:part=72T256520HFD3SB:revision=--/motherboard=0/memory-controller=1/dram-channel=2/dimm=3/rank=7  
[23:23:54] <tsnag>	 SRaid on nightshade is CRITICAL: NRPE: Unable to read output  
[23:24:23] <tsnag>	 s4 replag on z-dat-s4-a is CRITICAL: QUERY CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[23:24:33] <tsnag>	 Sun Grid Engine execd on ortelius is WARNING: NRPE: Unable to read output  
[23:24:43] <tsnag>	 MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[23:24:53] <tsnag>	 MySQL slave on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146)  
[23:24:54] <tsnag>	 Sun Grid Engine execd on wolfsbane is WARNING: NRPE: Unable to read output  
[23:25:24] <tsnag>	 Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds.  
[23:25:34] <tsnag>	 Sun Grid Engine execd on willow is WARNING: NRPE: Unable to read output  
[23:28:34] <tsnag>	 SMF on web.amaranth is CRITICAL: ERROR - maintenance:  svc:/application/jira:default  
[23:32:24] <tsnag>	 DiskSuite on damiana is CRITICAL: CRITICAL - submirror d52 of mirror d50 is Needs and submirror d32 of mirror d30 is Needs and submirror d12 of mirror d10 is Needs and submirror d22 of mirror d20 is Needs  
[23:34:33] <tsnag>	 MySQL slave on rosemary is OK: Uptime: 15495428  Threads: 10  Questions: 6879948764  Slow queries: 2250936  Opens: 328757  Flush tables: 6  Open tables: 4152  Queries per second avg: 443.998 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1700  
[23:34:43] <tsnag>	 s1 replag on rosemary is OK: QUERY OK: SELECT ts_rc_age() returned 1679.000000  
[23:42:03] <tsnag>	 CAM on hemlock is WARNING: WARNING - Storage ts-array5 (2 warnings): null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.A:S3: 42:, null :OSGi.com.sun.storage.cam.agent(device.2530):event.ProblemEvent.REC_BATTERY_NEAR_EXPIRATION.description:S17:Tray.85.Battery.B:S3: 42:  
[23:46:08] <DaBPunkt>	 @replag
[23:46:09] <tsbot>	 DaBPunkt: s1-rr-a: 18m 58s [-0.97 s/s]; s1-user: 18m 58s [-0.97 s/s]; s2-user: 1d 12h 36m 47s [-2.03 s/s]; s3-rr-a: 1m 6s [+0.01 s/s]; s3-user: 1m 6s [+0.01 s/s]; s6-rr-a: 11s [-0.00 s/s]; s6-user: 11s [+0.00 s/s]