HP DL370 G5 hpacucli shows 2 drives as Predictive Failure - safe to try a rebuild 1 by 1?

Question

I have:

Openfiler SAN ML370 G5 Smart Array 6400 slot 1 Array B Logical Drive 2 is a RAID5 array of 6 x 148GB 10k hot plug drives making 680GB no spare

Last Friday, the power went out, this machine was just plugged in to the wall and it went down hard. When it came back up, drive 1 and 4 of 0-5 drive changed to red flashing Fault light. The chart from the array guide shows that as "predictive failure has been received for this drive, replace as soon as possible". At the commandline hpacucli utility reports the same message; Predictive Failure. The activity lights flash normally. The fault chart says the drive hasn't 'failed' until the fault LED is on solid.

During all of this and now a week later the system stays up and no users reported any problems so far - all ESX hosts/VMs are using this SAN and are still working fine and I manually made a backup of everything on the array and new drives showed up today. So I can try a few things without too much effort, but I sure would like to just replace the drives and have rebuilding work if I'm careful.

Normally I would kind of assume that with just a predictive failure that I could get away with replacing them one at a time, letting them rebuild one at a time and be fine BUT when I run hpacucli I get the following output on the LD

Array: B Interface Type: Parallel SCSI Unused Space: 0 MB Status: OK

  Logical Drive: 2
     Size: 683.6 GB
     Fault Tolerance: RAID 5
     Heads: 255
     Sectors Per Track: 32
     Cylinders: 65535
     Stripe Size: 64 KB
     Status: OK
     Array Accelerator: Enabled
     Parity Initialization Status: Initialization Failed
     Unique Identifier: 600508B100104B39535153303250000F
     Disk Name: /dev/cciss/c0d1
     Mount Points: None
     Logical Drive Label: A01E9878P57820K9SQS02PBE24

So the Status is OK but the Parity Initialization is what has me spooked there. Any guidance on a procedure to have a successful rebuild appreciated - or advice along the lines of "all data is suspect now anyways, just replace the bad drives, make a new array out of it and restore since you have backup" are fine also. I get that it's a risk no matter what. Should I restart before I attempt to replace anything?

Full hpacucli output at the bottom.

Seems like if that Predictive Failure is just SMART errors piling up, it would still have parity and rebuild, just maybe slowly?

Many thanks for any guidance, Peace!

---full hpacucli---

Array: B Interface Type: Parallel SCSI Unused Space: 0 MB Status: OK

  Logical Drive: 2
     Size: 683.6 GB
     Fault Tolerance: RAID 5
     Heads: 255
     Sectors Per Track: 32
     Cylinders: 65535
     Stripe Size: 64 KB
     Status: OK
     Array Accelerator: Enabled
     Parity Initialization Status: Initialization Failed
     Unique Identifier: 600508B100104B39535153303250000F
     Disk Name: /dev/cciss/c0d1
     Mount Points: None
     Logical Drive Label: A01E9878P57820K9SQS02PBE24

  physicaldrive 1:0
     SCSI Bus: 1
     SCSI ID: 0
     Status: OK
     Drive Type: Data Drive
     Interface Type: Parallel SCSI
     Transfer Mode: Ultra 3 Wide
     Size: 146.8 GB
     Transfer Speed: 160 MB/Sec
     Rotational Speed: 10000
     Firmware Revision: HPB8
     Serial Number: 3HY83F3Y00007442557Q
     Model: COMPAQ  BD14685A26
  physicaldrive 1:1
     SCSI Bus: 1
     SCSI ID: 1
     Status: Predictive Failure
     Drive Type: Data Drive
     Interface Type: Parallel SCSI
     Transfer Mode: Ultra 3 Wide
     Size: 146.8 GB
     Transfer Speed: 160 MB/Sec
     Rotational Speed: 10000
     Firmware Revision: HPB8
     Serial Number: 3HY8393700007345XU2M
     Model: COMPAQ  BD14685A26
  physicaldrive 1:2
     SCSI Bus: 1
     SCSI ID: 2
     Status: OK
     Drive Type: Data Drive
     Interface Type: Parallel SCSI
     Transfer Mode: Ultra 3 Wide
     Size: 146.8 GB
     Transfer Speed: 160 MB/Sec
     Rotational Speed: 10000
     Firmware Revision: HPB8
     Serial Number: 3HY9NWGY00007524BFV1
     Model: COMPAQ  BD14685A26
  physicaldrive 1:3
     SCSI Bus: 1
     SCSI ID: 3
     Status: OK
     Drive Type: Data Drive
     Interface Type: Parallel SCSI
     Transfer Mode: Ultra 3 Wide
     Size: 146.8 GB
     Transfer Speed: 160 MB/Sec
     Rotational Speed: 10000
     Firmware Revision: HPB8
     Serial Number: 3HY9PA1N00007523W3DP
     Model: COMPAQ  BD14685A26
  physicaldrive 1:4
     SCSI Bus: 1
     SCSI ID: 4
     Status: Predictive Failure
     Drive Type: Data Drive
     Interface Type: Parallel SCSI
     Transfer Mode: Ultra 3 Wide
     Size: 146.8 GB
     Transfer Speed: 160 MB/Sec
     Rotational Speed: 10000
     Firmware Revision: HPB8
     Serial Number: 3HY72WR9000075216UNS
     Model: COMPAQ  BD14685A26
  physicaldrive 1:5
     SCSI Bus: 1
     SCSI ID: 5
     Status: OK
     Drive Type: Data Drive
     Interface Type: Parallel SCSI
     Transfer Mode: Ultra 3 Wide
     Size: 146.8 GB
     Transfer Speed: 160 MB/Sec
     Rotational Speed: 10000
     Firmware Revision: HPB8
     Serial Number: 3HY9NT3F000075231R9V
     Model: COMPAQ  BD14685A26

Answer 1

100% safe ? no. No operation involving an array rebuild is fully safe, in particular with RAID 5. Will it work ? Probably.

Your problem is that a single read failure during the rebuild will cause the whole volume to fail. And you're going to do it twice, including once with a drive that is already having trouble.

Ideally, in such a situation you should take the system out of production, make a full backup, delete the RAID volume, change your disks, re-create the array and restore your backup.

If you really can't take that much downtime on that array, then you should try to change each disk separately and wait each time for the rebuild to be done but do NOT do that without having a full backup first and, if you intend to keep that system active during the rebuild, make sure you warn the users first that it is possible that they will lose all data after the last backup date and that you got their approval (after all, it's their data and they should be the ones deciding what risk they'd rather take: safe but show with downtime or unsafe but potentially with uninterrupted service).

Answer 2

Copy your data off of the array while you can or try to take a backup of the relevant data. You can power off (spin the drives down) and back on to see if you can jog the rebuild process. Replace one disk at a time. Watch for the "waiting for rebuild" status. If you see that, it means that the RAID 5 rebuild won't be able to continue (usually a read error on another drive in the array).

Parity initialization is detailed here and here.

Background RAID creation 
When you create a RAID 1, RAID 5, or RAID 6 logical drive, the Smart Array controller must build the 
logical drive within the array and initialize the parity before enabling certain advanced performance 
techniques. Parity initialization takes several hours to complete. The time it takes depends on the size of the 
logical drive and the load on the controller. The Smart Array controller creates the logical drive, initializing 
the parity whenever the controller is not busy. While the controller creates the logical drive, you can access 
the storage volume which has full fault tolerance. 

What enclosure are you using for these disks? MSA30?

Answer 3

I had the exact same problem on a DL380 G7 P410i RAID5 and just went through a very unpleasant experience with HP support where I got absolutely nowhere, dealing with several people (from India) who had no idea what was going on and who were also unwilling to escalate the issue. The reference material was equally poor at explaining this error condition.

Anyway, I have solved the problem. From what I can tell, the “rebuild” operation is not necessarily the same as the “parity initialization”. In my case I had a defective HDD having a lot of recovered read errors but not actually flagged as failed by the array. The question was of course, “If I change out this disk, will the logical drive fail because maybe the parity has not been properly written?” and also “Why the **** hasn’t the HP management agent reported anything wrong?”. Well after having backed everything up and pulled out the defective disk, the logical drive kept running. I put in the replacement disk and the logical drive status changed to “Recovery”. When recovery completed the logical drive status changed back to “OK” but the parity initialization was still failed. I’m not sure how it happened in the first place, but it might have been due to the errors on my defective disk, as suggested above.

So finally, I did a “ctrl slot=0 logicaldrive 1 modify raid=5” from the ACU cli and this restarted the parity initialization, which finally completed successfully.