[00:26:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:27:08] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [00:27:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [00:27:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [00:32:28] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [00:34:13] nacht ts [01:26:38] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:27:07] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [01:27:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [01:27:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [01:32:27] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [02:26:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:27:07] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [02:27:17] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [02:27:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [02:32:28] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [03:26:37] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:27:18] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [03:27:47] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [03:28:07] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [03:32:28] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [04:27:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:28:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [04:28:15] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [04:28:43] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [04:32:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [05:27:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:28:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [05:28:24] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [05:28:44] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [05:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [06:27:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:28:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [06:28:24] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [06:28:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [06:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [07:27:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:28:25] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [07:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [07:29:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [07:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [07:52:05] Free Memory on turnera is CRITICAL: CRITICAL - 3.9% (329544 kB) free! [07:54:05] Free Memory on turnera is WARNING: WARNING - 5.7% (474772 kB) free! [07:58:06] Free Memory on turnera is OK: OK - 7.2% (603188 kB) free. [08:27:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [08:29:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [08:29:26] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [08:33:33] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [09:27:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [09:30:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [09:30:24] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [09:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [09:43:37] @replag [09:43:38] Merlissimo: s1-rr-a: error; s1-rr-a-wd: error; s1-user-wd: 1m 4s [+0.00 s/s]; s2-user-wd: 1m 4s [+0.00 s/s]; s3-rr-a: 11h 3s [+1.00 s/s]; s3-user: 11h 3s [+1.00 s/s]; s4-user-wd: 1m 4s [+0.00 s/s]; s5-user-wd: 1m 4s [+0.00 s/s] [09:43:39] Merlissimo: s6-rr-a: 10m 5s [+0.00 s/s]; s6-user: 10m 5s [+0.00 s/s]; s7-rr-a: 10m 18s [-0.63 s/s]; s7-user: 10m 18s [-0.63 s/s] [10:27:33] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [10:30:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [10:30:25] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [10:33:33] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [11:27:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [11:30:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [11:30:25] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [11:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [12:15:05] Free Memory on turnera is WARNING: WARNING - 5.6% (469488 kB) free! [12:22:06] Free Memory on turnera is OK: OK - 7.1% (591736 kB) free. [12:27:34] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [12:30:24] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [12:31:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [12:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [13:10:05] Free Memory on turnera is WARNING: WARNING - 6.1% (514976 kB) free! [13:15:05] Free Memory on turnera is OK: OK - 8.2% (684688 kB) free. [13:27:43] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:28:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [13:30:26] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [13:31:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [13:33:33] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [14:27:44] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:28:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [14:30:06] Free Memory on turnera is WARNING: WARNING - 6.4% (532716 kB) free! [14:30:25] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [14:31:15] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [14:32:06] Free Memory on turnera is OK: OK - 7.6% (639016 kB) free. [14:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [15:27:44] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [15:30:25] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [15:32:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [15:33:33] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [16:00:04] Andre Koopal * [Toolserver-l] mdbtools [16:24:05] Free Memory on turnera is WARNING: WARNING - 6.6% (549096 kB) free! [16:27:43] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:28:53] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [16:30:25] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [16:33:06] Free Memory on turnera is OK: OK - 7.8% (656772 kB) free. [16:33:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [16:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [16:52:05] Free Memory on turnera is WARNING: WARNING - 6.7% (560968 kB) free! [16:59:04] Platonides * Re: [Toolserver-l] mdbtools [17:06:05] Free Memory on turnera is OK: OK - 7.7% (648344 kB) free. [17:07:55] @replag [17:07:55] liangent: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 50m 34s [+0.05 s/s]; s1-user-c: 30s [-0.00 s/s]; s1-user-wd: 33s [-0.00 s/s]; s2-user-wd: 33s [-0.00 s/s]; s3-rr-a: 18h 24m 21s [+1.00 s/s]; s3-user: 18h 24m 21s [+1.00 s/s] [17:07:56] liangent: s4-user-wd: 33s [-0.00 s/s]; s5-user-wd: 33s [-0.00 s/s] [17:08:05] @replag all [17:08:05] liangent: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 50m 38s [+0.43 s/s]; s1-user-c: 17s [-1.39 s/s]; s1-user-wd: 38s [+0.53 s/s]; s2-user: 13s [+0.00 s/s]; s2-user-c: 1s [-]; s2-user-wd: 38s [+0.53 s/s] [17:08:05] liangent: s3-rr-a: 18h 24m 30s [+0.96 s/s]; s3-user: 18h 24m 30s [+0.97 s/s]; s4-rr-a: 1s [-]; s4-user: 1s [-]; s4-user-wd: 27s [-0.65 s/s]; s5-rr-a: 0s [-0.00 s/s]; s5-user: 0s [-0.00 s/s]; s5-user-c: 1s [-] [17:08:06] liangent: s5-user-wd: 38s [+0.54 s/s]; s6-rr-a: 0s [-0.02 s/s]; s6-user: 0s [-0.02 s/s]; s7-rr-a: 1s [-0.02 s/s]; s7-user: 1s [-0.02 s/s] [17:15:33] Platonides: thanks for looking at least [17:16:18] Brooke: Do you have a db corruption script? Or how are you figuring out the corruptions? [17:27:44] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [17:30:33] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [17:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [17:34:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [18:05:05] Free Memory on turnera is WARNING: WARNING - 6.6% (555644 kB) free! [18:07:05] Free Memory on turnera is OK: OK - 7.2% (602596 kB) free. [18:27:44] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [18:31:25] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [18:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [18:35:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [19:20:30] nighty o/ [19:27:44] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [19:31:25] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [19:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [19:35:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [20:27:44] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [20:31:25] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [20:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [20:35:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:27:44] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:28:52] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [21:31:25] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [21:33:06] Free Memory on turnera is WARNING: WARNING - 6.3% (527908 kB) free! [21:33:33] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [21:35:05] Free Memory on turnera is OK: OK - 7.8% (657340 kB) free. [21:35:13] @replag [21:35:14] Vacation9: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 24m 40s [-0.10 s/s]; s1-user-wd: 48s [+0.00 s/s]; s2-user-wd: 48s [+0.00 s/s]; s3-rr-a: 22h 51m 39s [+1.00 s/s]; s3-user: 22h 51m 39s [+1.00 s/s]; s4-user-wd: 48s [+0.00 s/s] [21:35:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [21:35:15] Vacation9: s5-user-wd: 49s [+0.00 s/s] [21:35:44] @replag [21:35:45] Vacation9: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 24m 24s [-0.52 s/s]; s1-user-wd: 54s [+0.19 s/s]; s2-user-wd: 54s [+0.19 s/s]; s3-rr-a: 22h 52m 11s [+1.02 s/s]; s3-user: 22h 52m 11s [+1.02 s/s]; s4-user-wd: 54s [+0.19 s/s] [21:35:46] Vacation9: s5-user-wd: 54s [+0.16 s/s] [21:58:15] @replag [21:58:15] Danny_B|backup: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 16m 32s [-0.35 s/s]; s1-user-wd: 33s [-0.02 s/s]; s2-user-wd: 34s [-0.01 s/s]; s3-rr-a: 23h 14m 41s [+1.00 s/s]; s3-user: 23h 14m 41s [+1.00 s/s]; s4-user-wd: 34s [-0.01 s/s] [21:58:16] Danny_B|backup: s5-user-wd: 34s [-0.01 s/s] [22:27:44] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:29:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [22:31:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [22:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [22:35:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [22:47:57] @replag all [22:47:58] Merlissimo: s1-rr-a: error; s1-rr-a-wd: error; s1-user: 1m 50s [-0.30 s/s]; s1-user-c: 1s [-0.00 s/s]; s1-user-wd: 1m 41s [+0.02 s/s]; s2-user: 7s [-0.00 s/s]; s2-user-c: 1s [-]; s2-user-wd: 1m 41s [+0.02 s/s] [22:47:59] Merlissimo: s3-rr-a: 1d 4m 23s [+1.00 s/s]; s3-user: 1d 4m 23s [+1.00 s/s]; s4-rr-a: 1s [-]; s4-user: 1s [-]; s4-user-wd: 1m 41s [+0.02 s/s]; s5-rr-a: 2s [+0.00 s/s]; s5-user: 2s [+0.00 s/s]; s5-user-c: 1s [-] [22:48:00] Merlissimo: s5-user-wd: 1m 41s [+0.02 s/s]; s6-rr-a: 1s [+0.00 s/s]; s6-user: 1s [+0.00 s/s]; s7-rr-a: 9s [+0.00 s/s]; s7-user: 9s [+0.00 s/s] [23:09:10] s3 replag halted? [23:27:44] Environment IPMI on wolfsbane is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:29:06] Environment IPMI on thyme is CRITICAL: CHECK_NRPE: Socket timeout after 30 seconds. [23:31:34] RAID on rosemary is CRITICAL: ERROR - TOTAL: 2: FAILED: 0: DEGRADED: 1 [23:33:34] MySQL on thyme is CRITICAL: Cant connect to MySQL server on thyme (146) [23:35:14] MySQL on z-dat-s4-a is CRITICAL: Cant connect to MySQL server on z-dat-s4-a (146) [23:59:05] Free Memory on turnera is CRITICAL: CRITICAL - 4.7% (397376 kB) free!