3

I've got a server running Debian Squeeze and a 3x 500 GB-drive RAID5 system which I haven't set up myself. When booting, the status of one partition in the RAID-array seems to be bad.

md: bind<sda2>
md: bind<sdc2>
md: bind<sdb2>
md: kicking non-fresh sda2 from array!
md: unbind<sda2>
md: export_rdev(sda2)
raid5: device sdb2 operational as raid disk 1
raid5: device sdc2 operational as raid disk 2
raid5: allocated 3179kB for md1
1: w=1 pa=0 pr=3 m=1 a=2 r=3 op1=0 op2=0
2: w=2 pa=0 pr=3 m=1 a=2 r=3 op1=0 op2=0
raid5: raid level 5 set md1 active with 2 out of 3 devices, algorithm 2
RAID5 conf printout:
 --- rd:3 wd:2
 disk 1, o:1, dev:sdb2
 disk 2, o:1, dev:sdc2
md1: detected capacity change from 0 to 980206485504
 md1: unknown partition table

mdstat also tells me the partition is missing:

Personalities : [raid1] [raid6] [raid5] [raid4] 
md1 : active raid5 sdb2[1] sdc2[2]
      957232896 blocks level 5, 64k chunk, algorithm 2 [3/2] [_UU]

md0 : active raid1 sda1[0] sdc1[2](S) sdb1[1]
      9767424 blocks [2/2] [UU]

When running sudo mdadm -D, the partition shows up as removed, and the array as degraded.

/dev/md1:
        Version : 0.90
  Creation Time : Mon Jun 30 00:09:01 2008
     Raid Level : raid5
     Array Size : 957232896 (912.89 GiB 980.21 GB)
  Used Dev Size : 478616448 (456.44 GiB 490.10 GB)
   Raid Devices : 3
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Thu Aug 11 16:58:50 2011
          State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 03205c1c:cef34d5c:5f1c2cc0:8830ac2b
         Events : 0.275646

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2

/dev/md0:
        Version : 0.90
  Creation Time : Mon Jun 30 00:08:50 2008
     Raid Level : raid1
     Array Size : 9767424 (9.31 GiB 10.00 GB)
  Used Dev Size : 9767424 (9.31 GiB 10.00 GB)
   Raid Devices : 2
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Thu Aug 11 17:21:20 2011
          State : active
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

           UUID : f824746f:143df641:374de2f8:2f9d2e62
         Events : 0.93

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1

       2       8       33        -      spare   /dev/sdc1

However, md0 seems to be ok. So, what does all this tell me? Can the disk be faulty even though md0 is working? If not, can I just re-add /dev/sda2 to the md1 array to solve the problem?

3 Answers 3

3

The R in RAID stands for Redundant.

RAID 5 is N+1 redundant: If you lose one disk you're at N -- The system will keep operating fine as long as you don't lose another one. If you lose a second disk you are now at N-1 and your universe collapses (or at the very least you lose lots of data).

Like SvenW said, replace the disk AS SOON AS POSSIBLE (Follow your distribution's instructions for replacing disks in md RAID arrays, and for God's sake make sure you replace the correct disk! Pulling out one of the active disks will really screw up your day.)
Also be aware that when you replace a disk in a RAID 5 there is a lot of resulting disk activity as the new drive is rebuilt (lots of reads on the old disks, lots of writes on the new one). This has two major implications:

  1. Your system will be slow during the rebuild.
    How slow depends on your disks and disk I/O subsystem.

  2. You may lose another disk during/shortly after the rebuild.
    (All that disk I/O sometimes triggers enough errors from another drive that the controller declares it "bad").

The chances of #2 increase as you have more disks in your array, and follows the standard "bathtub curve" of hard drive mortality. This is part of why you should have a backup, and one of the many reasons you hear the mantra "RAID is not a backup" repeated so often on ServerFault.

6
  • All right. When a drive is kicked from an array and shows up as "removed", does this always mean the drive is faulty? An another thing; why is the status removed instead of failed? No, I'm not arguing with any of you regarding replacement of the drive, but not one of you has mentioned any other option than the disk is faulty.
    – nip3o
    Aug 11, 2011 at 16:34
  • 1
    Get your RAID array rebuilt with a good drive to restore redundancy ASAP, then worry about whether or not the other drive is good.
    – MikeyB
    Aug 11, 2011 at 17:57
  • @nico - you do not / cannot know the fault status of the disk (why it was removed from the array) -- The previous admin could have taken it out for some reason or no reason. Presumably you care about the data on your RAID, so it behooves you to look skeptically upon the disk and assume it's faulty and must be replaced. You can run it on a disk exerciser later, but your priority should always be data integrity.
    – voretaq7
    Aug 11, 2011 at 17:58
  • There aren't many absolutes in RAID - but two are get a good backup just in case before you rebuild the array, and always put in a new drive. How else will you know if the port/controller has issues? If you must - test / clean / sanitize / reuse the drive later, but you want your "best team on the field" at this point.
    – bmike
    Aug 11, 2011 at 20:43
  • As said, I am not questioning the fact that the disc needs replacement. The data was being backed up before even posting the question, and the drive is being replaced soon. Thanks for your concerns though. :) I ran a short SMART-test on the kicked drive (maybe not too clever, but as said the data is backed up onto an external drive), which indicated several read errors, so it's most certainly a drive hardware issue.
    – nip3o
    Aug 11, 2011 at 20:54
3

Keeping the array working with a broken disk is the exact purpose of a RAID5. It keeps redundancy informations so you can lose one disk and still don't have data loss.

I would recommend to replace the disk as soon as possible because if you lose another disk, all your data will be gone.

2
  • I disagree, the purpose of RAID 5 is to prevent data loss in the event of a disk failure. Keeping a raid running with a disk out runs a much higher risk of additional disk failure than running with no raid at all.
    – Ashigore
    Oct 16, 2014 at 13:38
  • @Ashigore: You disagree with what? Your first sentence states the same stuff I said. The second part is flat out wrong - having two (or more disks) running in an incomplete RAID5 will not alter the risk of another disk failure when compared to the same number of disks in a non-RAID scenario with similar access patterns. However, the impact of a second disk failure in a RAID5 is more severe then with standalone disks and during recovery, the risk of a failure is also higher due to the high work load, but that can't be avoided.
    – Sven
    Oct 16, 2014 at 14:13
1

Even though /dev/sda1 appears to be working fine in md0 now, the fact that the other partition on the same disk (sda2) is faulty bodes ill for the health of the drive. I must concur with the other opinions already expressed here: replace the sda drive immediately.

Of course, that means you will need to mdadm --fail and mdadm --remove partition sda1 from array md0, even though it appears to be fine right now. And when you install the replacement drive, you will need to ensure that its partitions are at least as large as those on the old drive, so that its partitions can be properly added to the md0 and md1 arrays.

1
  • md on Linux gives software RAID a bad name -- that's way too much work for lazy ol' me :-)
    – voretaq7
    Aug 11, 2011 at 16:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .