[00:01:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:06:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:11:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:16:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:21:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:26:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:31:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:36:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:41:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:46:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:46:23] <wikibugs_>	 fundraising-tech-ops, Operations, ops-codfw: frdb2001 RAID disk failure - https://phabricator.wikimedia.org/T196251#4251134 (Jgreen)
[00:51:09] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[00:56:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[01:01:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[01:06:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[01:11:29] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[01:16:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[01:21:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[01:26:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[01:31:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[01:36:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]
[01:41:19] <icinga-wm>	 PROBLEM - check_raid on frdb2001 is CRITICAL: CRITICAL: HPSA [P420i/slot0: OK, log_1: 3.3TB,RAID1+0 OK, phy_1I:1:11: Predictive Failure]