Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have MSA20 with one disk array on 12 disks and 3 LUNs on it (each raid 5). A few days ago one disk in one of the LUNs was failed and I replaced it. But raid5 recovering failed at 13% and I see in ADU report that one of the disk has "Errors Logged = 5566" and according SCSI specifications it is URE (Sense Code=0x11, Qualifier=0x00). In serial log I also see URE error. It seems that Raid5 can't be rebuilt because of this. So I have a few questions:

  1. Is there a way to recover raid5 still?

  2. If I leave new disk that was replaced and remove disk with URE, will other LUNs be destroyed or just failed LUN? If all LUNs will fail what is the sense to make each LUN with own raid on one disk group array if 2 failed disk can destroy all?

  3. As I understand the preferred way is to create one disk array for one LUN in future and not one array with few LUNs?

Thanks.

share|improve this question
    
It is HP MSA 1500cs with HP MSA20 enclosure so controller automatically starts to rebuild failed LUN when hot swappable failed disk replaced. –  Andrey Jun 9 '12 at 9:57
    
Thank you for the answer, I'll try, but I'm not sure that 1500cs will ask to re-enable units. Additional problem that all disk are marked as OK even that has URE error. So RAID 5 volume can't get full recover. It starts recovering but at 15% restart it from scratch due to Unrecovered Read Error on other disk. –  Andrey Jun 9 '12 at 11:44
    
So LUN's tolerance is not compromised yet, LUNs are in interim recovery mode now without one drive and works in degraded mode. –  Andrey Jun 9 '12 at 11:47
    
Is the original failed disk already gone? –  Dmitri Chubarov Jun 9 '12 at 11:53
    
I think no, the led was yellow, but I reinserted it and it started to rebuild (with the same 15% bad result). Actually I don't like MSA20. It can lose disks but in few cases if "failed" disk will be reseated LUN rebuilds fine. But not in this case unfortunately. –  Andrey Jun 9 '12 at 12:12
show 1 more comment

2 Answers

1) It is very unlikely that you'll be able to recover this particular array. RAID is not backup. This is one of the many reasons you need backups.

2) It depends how the LUNs are set up. If you have one RAID 5 array with all 12 disk that is separated into 3 logical units, then since the array is gone, all its logical units are gone. If you have three separate RAID 5 arrays each with 4 disks, then only the array containing these two disks is gone, and the other arrays (and hence their logical units) will be fine.

3) It largely depends on what you want to do. There may be good reasons to have separate arrays on separate disks. For example, you may want to prevent a heavily loaded array from slowing down other arrays. If the arrays are on the same physical disks, you can't do this. Or you may want to allow a heavily loaded array to be able to get all the bandwidth of all the disks. If you have separate arrays on separate disks, you can't do that.

And there are also reasons you might want to put multiple logical units on the same array. You may want to isolate filesystems so that filling one up won't fill up the other.

If you put all the logical units on one array, you lose less space. A single RAID 5 array on 12 1TB disks gives you 11TB usable, divided into three equal portions, that's 3.6TB each. If you create three separate arrays each with 4 1TB disks, that's 3TB each. So you'd trade off size to get the extra reliability.

The specifics of what flexibility you have and what affects that has depends on the specifics of your controller.

And, some advice for the future:

  1. Consider RAID 6. It can tolerate the failure of two drives.

  2. Make 100% sure your arrays are tested regularly and that failed drives are replaced promptly. This will dramatically reduce the chance of a drive failure during a degraded state.

  3. RAID is not backup. Keep regular backups to a physically-separate device.

If you have data on there that's not backed up, try to recover as much of it as possible immediately. However, if you can't even get the array to mount, professional recovery is your only hope.

share|improve this answer
    
Thank you for the answer. I have one disk array with 12 disks but this array is divided on 3 LUNS each with raid5. So I sut want to know if I lost one LUN due do 2 disk failures are other two LUNs safe if I remove two disks of failed LUN? –  Andrey Jun 9 '12 at 6:48
    
If you have one 12-disk array, and you lost two disks, then the array is lost. The nature of a RAID 5 array is that if you lose two of its disks, the array is lost. (Had you made three arrays, each with 4 disks, then only one array would be lost.) –  David Schwartz Jun 9 '12 at 6:52
    
According to my experience msa20 is not very good sata enclosure unfortunately. It can lose normal disks and have other issues even with tha latest firmware. Also this msa20 is already for backups, so there is no really critical data on it, I'll consider raid 6, thanks for the advices. –  Andrey Jun 9 '12 at 6:53
    
Glad to hear it's only backups that are at risk. Sorry I don't have better news. –  David Schwartz Jun 9 '12 at 6:58
1  
Since every LUN is spread over the 12 disks, the other LUNs are spread over the failed disks. They are likely failed as well. (But it can't hurt to check.) –  David Schwartz Jun 9 '12 at 7:24
show 9 more comments

1) Is there a way to recover?

  • Power down the array.
  • Reseat each disk one by one.
  • Power up the array again. See if any of the failed disks come online.

Follow the Maintenance and Administration Guide for your controller on handling failed volumes.

While the array is powered down check the S.M.A.R.T data on the failed drives. The drives may be marked failed but still readable for instance if they are running low on reserved sectors for data relocation.

On MSA1500cs a recovery procedure is explained chapter 9 of Maintenance and Service Guide:

If fault tolerance is compromised, inserting replacement hard drives does not improve the condition of the logical unit. The procedure to re-enable or accept a LUN that is unresponsive is performed in the Array Configuration Utility (ACU) or the MSA Command Line Interface (MSA-CLI).

  1. Stop all I/O activity.

  2. Turn off the system

  3. Remove and then reinsert all hard rives and controllers.

  4. Turn the system on

    5.1. Check if the following messages are shown on the LCD display:

     02 ENABLE VOLUME <n>?  '<'=NO, '>'=YES
     04 ENABLE VOLUMES ? '<'=NO, '>'=YES
    

    5.2. If using the ACU: Select Re-enable Failed Logical Drive

    5.3. If using the MSA-CLI: Enter accept units to enable all faulted LUNs.

I find MSA-CLI to be the most convenient way to manage the array.

2) If I leave new disk that was replaced and remove disk with URE, will other LUNs be destroyed or just failed LUN?

That depends on the logical organization of your arrays.

Here is an example of a disk shelf split into 2 disk arrays. One disk array has 4 Logical volumes on it and another has 2 logical volumes.

# hpacucli controller csn=sga0xxxx0f array all show
MSA1500 CS in MSA1500
array A
  logicaldrive 1 (2.0 TB, RAID 6 (ADG), OK)
  logicaldrive 10 (2.0 TB, RAID 5, OK)
  logicaldrive 11 (2.0 TB, RAID 5, OK)
  logicaldrive 12 (1.3 TB, RAID 5, OK)
array B
  logicaldrive 2 (2.0 TB, RAID 6 (ADG), OK)
  logicaldrive 4 (2.0 TB, RAID 6 (ADG), OK)

A failure of 2 disks in array A will result in failure of logical drives 10, 11 and 12. Logical drive 1 will not fail since it is a RAID6.

3) As I understand the preferred way is to create one disk array for one LUN in future and not one array with few LUNs?

It depends. Spreading LUNs thin may improve performance since the load is spread on all the drives. Separating LUNs into different arrays would protect from interference between LUNs both in terms of performance and reliability, but at a cost. It is often easier to cut LUNs of arbitrary size from a single large pool than to have multiple small pools.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.