Dell PERC 5 - RAID-10 keeps rebuilding drive 2 every day

Question

I have a Dell PowerEdge 2950 with this card: RAID bus controller [0104]: Dell PowerEdge Expandable RAID controller 5 [1028:0015] and six disks in a RAID-10.

I replaced drive 2, because it didn't show up, and then it started to rebuild itself:

root@backup01:~# megaraidsas-status
-- Arrays informations --
-- ID | Type | Size | Status
a0d0 | RAID 10 | 5587GiB | DEGRADED

-- Disks informations
-- ID | Model | Status | Warnings
a0e8s0 | ATA ST2000DM001-9YN1 1863GiB | online | errs: media:0 other:5393
a0e8s1 | ATA ST2000DM001-9YN1 1863GiB | online | errs: media:0 other:5394
a0e8s2 | ATA ST2000DM001-1E61 1863GiB | rebuild | errs: media:0 other:99
a0e8s3 | ATA ST2000DM001-9YN1 1863GiB | online | errs: media:0 other:5393
a0e8s4 | ATA ST2000DM001-9YN1 1863GiB | online | errs: media:0 other:5393
a0e8s5 | ATA ST2000DM001-9YN1 1863GiB | online | errs: media:0 other:5393

The rebuild finishes, then the virtual drive becomes optimal, and drive 2 goes online. Then once a day, drive 2 acts like it's been removed, and the rebuild starts all over again. How do I make this once a day rebuild stop?

Event Description: Removed: PD 02(e1/s2)
Event Description: Removed: PD 02(e1/s2) Info: enclPd=08, scsiType=0, portMap=04, sasAddr=1221000002000000,0000000000000000
Event Description: State change on VD 00/0 from OPTIMAL(3) to DEGRADED(2)
Event Description: VD 00/0 is now DEGRADED1
Event Description: State change on PD 02(e1/s2) from ONLINE(18) to FAILED(11)
Event Description: State change on PD 02(e1/s2) from FAILED(11) to UNCONFIGURED_BAD(1)
Event Description: Background Initialization failed on VD 00/0
Event Description: Inserted: PD 02(e1/s2)
Event Description: Inserted: PD 02(e1/s2) Info: enclPd=08, scsiType=0, portMap=04, sasAddr=1221000002000000,0000000000000000
Event Description: PD 02(e1/s2) is not a certified drive
Event Description: State change on PD 02(e1/s2)
Event Description: State change on PD 02(e1/s2) from UNCONFIGURED_GOOD(0) to OFFLINE(10) from UNCONFIGURED_BAD(1) to UNCONFIGURED_GOOD(0)
Event Description: Rebuild automatically started on PD 02(e1/s2)
Event Description: State change on PD 02(e1/s2) from OFFLINE(10) to REBUILD(14)

Answer 1

To give you a preliminary.. It sounds to me like this is a double data fault/puncture, which means that you have bad blocks which are propogated within the RAID. This can cause some really really strange issues in my experience. You can read up on it here : http://www.dell.com/support/Article/us/en/04/438291/en

Any chance we can get an output of your RAID controller log? Do you have the dell open manage server administrator packages installed? If not that's the best place to start. It looks like you're running some flavor of linux, you can download the OMSA packages : http://linux.dell.com/repo/hardware/omsa.html.

Once you have open manage you can export a RAID controller log using this command : omconfig storage controller action=exportlog controller=id, where the controller is the ID of your controller (should be 0 unless you have more than one RAID controller). Once we have that output we'll be able to confirm what's going on. (hopefully).

Only one last thing, your expandable RAID 5, do you know if that's a PERC5i, a SAS5, or another model?

Hope to be of assistance!