Hi I am new to VMware. I took on a client about a year ago that has an ESXI 4.1 server with 4 virtual machines. These virtual machines include the domain controller/file server, email server, database server, and a standalone windows 7 installation. The server is a PowerEdge T410 with a 3 drive Raid 5 array. While monitoring the server a couple of weeks ago, I noticed that the drive in bay 0 in the raid 5 array had a drive fault and it was blinking amber. I went and bought two replacement drives from serversupply.com. The replacement drives were the same Seagate Cheetah ST3600057SS 600GB 15000RPM drives that are in the server now. The only difference is the firmware. The drives in the server are Fw: ES68 and the drives from server supply are ES62. I believe the firmware on the failed drive was ES64. To make a long story short, I hot swapped the bad drive with one of drives that I received from server supply. It began to rebuild. It was still rebuilding a week later. I tried to push Dell Open Manage to the server via VMware CLI to see what was happening and it would just sit there saying In Progress in vSphere client. I decided to shut down the virtual machines and reboot the server. While the server was rebooting, I decided to go into the expandable raid configuration in the bios to take a look at the raid. It said that the raid was degraded. I then did dell diagnostics on the drive in bay 0 and it reported numerous medium block errors. I decided to try and go back into esxi 4.1 and get the users back up an running because they had already been down for a while. When ESXI 4.1 started to load it got to "cbt loaded successfully" and then would freeze and would not move any further. In terms of getting the users back up and running I was stuck in the water. I decided to pull the drive in bay 0 and then ESXI 4.1 loaded correctly working off of the other two drives. I figured that that drive was bad so I hot swapped the drive in bay 0 again and then rebooted. I went into the raid configuration and the rebuild on the new drive began. After an hour or so, the rebuild completed and the raid array went to OPTIMAL state. I then tried to boot into ESXI 4.1 and it got stuck at "cbt loaded successfully" again. I then decided to pull the drive in bay 0 again and ESXI 4.1 loaded successfully again. After ESXI 4.1 booted up, I put the drive back into bay 0 thinking that it would rebuild again. I got the virtual machines back up and running and got the users back to work but the drive in bay 0 still shows rebuilding in the configuration on vSphere client. It has been in this rebuilding state for the past 5 days. The configuration states that the array is degraded again and I still can't get Dell OpenManage installed. It still just sits on In Progress.
I need to get the raid 5 in OPTIMAL state again so that we can have 3 working drives. I know that if I lose another drive before I do that then I am toast. Anyone have any ideas on what I may be doing wrong? Any ideas that may help me get this array back to optimal state without losing my vms or experiencing anymore downtime?