MSA20 RAID5 recovery failure due to URE on another disk

Question

I have MSA20 with one disk array on 12 disks and 3 LUNs on it (each raid 5). A few days ago one disk in one of the LUNs was failed and I replaced it. But raid5 recovering failed at 13% and I see in ADU report that one of the disk has "Errors Logged = 5566" and according SCSI specifications it is URE (Sense Code=0x11, Qualifier=0x00). In serial log I also see URE error. It seems that Raid5 can't be rebuilt because of this. So I have a few questions:

1. Is there a way to recover raid5 still?

2. If I leave new disk that was replaced and remove disk with URE, will other LUNs be destroyed or just failed LUN? If all LUNs will fail what is the sense to make each LUN with own raid on one disk group array if 2 failed disk can destroy all?

3. As I understand the preferred way is to create one disk array for one LUN in future and not one array with few LUNs?

Thanks.

Answer 1

1) It is very unlikely that you'll be able to recover this particular array. RAID is not backup. This is one of the many reasons you need backups.

2) It depends how the LUNs are set up. If you have one RAID 5 array with all 12 disk that is separated into 3 logical units, then since the array is gone, all its logical units are gone. If you have three separate RAID 5 arrays each with 4 disks, then only the array containing these two disks is gone, and the other arrays (and hence their logical units) will be fine.

3) It largely depends on what you want to do. There may be good reasons to have separate arrays on separate disks. For example, you may want to prevent a heavily loaded array from slowing down other arrays. If the arrays are on the same physical disks, you can't do this. Or you may want to allow a heavily loaded array to be able to get all the bandwidth of all the disks. If you have separate arrays on separate disks, you can't do that.

And there are also reasons you might want to put multiple logical units on the same array. You may want to isolate filesystems so that filling one up won't fill up the other.

If you put all the logical units on one array, you lose less space. A single RAID 5 array on 12 1TB disks gives you 11TB usable, divided into three equal portions, that's 3.6TB each. If you create three separate arrays each with 4 1TB disks, that's 3TB each. So you'd trade off size to get the extra reliability.

The specifics of what flexibility you have and what affects that has depends on the specifics of your controller.

And, some advice for the future:

1. Consider RAID 6. It can tolerate the failure of two drives.

2. Make 100% sure your arrays are tested regularly and that failed drives are replaced promptly. This will dramatically reduce the chance of a drive failure during a degraded state.

3. RAID is not backup. Keep regular backups to a physically-separate device.

If you have data on there that's not backed up, try to recover as much of it as possible immediately. However, if you can't even get the array to mount, professional recovery is your only hope.

Answer 2

1) Is there a way to recover?

• Power down the array.
• Reseat each disk one by one.
• Power up the array again. See if any of the failed disks come online.

Follow the Maintenance and Administration Guide for your controller on handling failed volumes.

While the array is powered down check the S.M.A.R.T data on the failed drives. The drives may be marked failed but still readable for instance if they are running low on reserved sectors for data relocation.

On MSA1500cs a recovery procedure is explained chapter 9 of Maintenance and Service Guide:

If fault tolerance is compromised, inserting replacement hard drives does not improve the condition of the logical unit. The procedure to re-enable or accept a LUN that is unresponsive is performed in the Array Configuration Utility (ACU) or the MSA Command Line Interface (MSA-CLI).

1. Stop all I/O activity.

2. Turn off the system

3. Remove and then reinsert all hard rives and controllers.

4. Turn the system on

   5.1. Check if the following messages are shown on the LCD display:

    02 ENABLE VOLUME <n>?  '<'=NO, '>'=YES
    04 ENABLE VOLUMES ? '<'=NO, '>'=YES
   

   5.2. If using the ACU: Select Re-enable Failed Logical Drive

   5.3. If using the MSA-CLI: Enter accept units to enable all faulted LUNs.

I find MSA-CLI to be the most convenient way to manage the array.

2) If I leave new disk that was replaced and remove disk with URE, will other LUNs be destroyed or just failed LUN?

That depends on the logical organization of your arrays.

Here is an example of a disk shelf split into 2 disk arrays. One disk array has 4 Logical volumes on it and another has 2 logical volumes.

# hpacucli controller csn=sga0xxxx0f array all show
MSA1500 CS in MSA1500
array A
  logicaldrive 1 (2.0 TB, RAID 6 (ADG), OK)
  logicaldrive 10 (2.0 TB, RAID 5, OK)
  logicaldrive 11 (2.0 TB, RAID 5, OK)
  logicaldrive 12 (1.3 TB, RAID 5, OK)
array B
  logicaldrive 2 (2.0 TB, RAID 6 (ADG), OK)
  logicaldrive 4 (2.0 TB, RAID 6 (ADG), OK)

A failure of 2 disks in array A will result in failure of logical drives 10, 11 and 12. Logical drive 1 will not fail since it is a RAID6.

3) As I understand the preferred way is to create one disk array for one LUN in future and not one array with few LUNs?

It depends. Spreading LUNs thin may improve performance since the load is spread on all the drives. Separating LUNs into different arrays would protect from interference between LUNs both in terms of performance and reliability, but at a cost. It is often easier to cut LUNs of arbitrary size from a single large pool than to have multiple small pools.