[14:33:48] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2575752 (10jcrespo) Right now, es2004 still shows:  ``` CRITICAL: 1 failed LD(s) (Degraded) ```  ``` Enclosure Device ID: 32 Slot Number: 10 Drive's position: DiskGroup: 0, Span:...
[15:58:16] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2575934 (10Papaul) @jcrespo the disk is a brain new disk that was in a static plastic bag nerve used .
[16:02:25] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2575943 (10jcrespo) I believe you, I am just copying and pasting: "Firmware state: Failed"  Either you changed the wrong disk- I *do not* believe that, the serial number seems di...
[16:05:03] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2575954 (10jcrespo) There is a more likely possibility- the controller has a problem with that particular port- the controler before didn't failed as usual, some if its informati...
[16:10:07] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2561047 (10Volans) @jcrespo true but now it returns immediately, so maybe it was just not recognized? Maybe you could try to unplug it and plug it again.  ```  es2004  0 ~$ time...
[16:14:22] <volans>	 FYI jynus: on es2004 ganglia-monitor seems to not being able to be started, after each puppet run fails and the respawing get blocked 
[16:15:22] <jynus>	 volans, wait, we were about to restart that
[16:15:23] <jynus>	 but
[16:15:31] <jynus>	 doesn't ganglia do that
[16:15:37] <jynus>	 starting every time?
[16:15:47] <jynus>	 because it is not really a deamon
[16:16:01] <jynus>	 in any case, we are about to restart that for hardware maintenance
[16:16:14] <volans>	 [7628978.127950] init: ganglia-monitor respawning too fast, stopped
[16:16:39] <volans>	 you can check dmesg or syslog
[16:17:00] <volans>	 ok, no problem, I was not doing anything, just a quick look at it ;)
[16:41:52] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2576153 (10jcrespo) Yes, it seems that it may need a reboot + configuration, that is the working thesis now.
[18:04:30] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2576561 (10jcrespo) I've put it down and downtime'd it for a day, @papaul feel free to start it and do anything with it configuration-wise (it is not urgent).
[20:41:06] <wikibugs>	 10DBA, 06Operations, 10ops-codfw: es2004 has a dead disk, but it is not under warranty - https://phabricator.wikimedia.org/T143220#2577372 (10Papaul) @jcrespo  the Raid controller is showing that it is saying the disk what you need to do is to put the new disk in the Raid10 see image below.   {F4394285}  {F4...