2

One of the drives in my 3-disk raid5 array is starting to show read errors and SMART warnings. Not enough to get it kicked out of the array (as faulty), but it affects performance and will probably go bad (worse). I obviously want to replace this drive.

Now the question is if I run this: (sdc is the broken drive and sdd is the new one):

mdadm /dev/md0 -a /dev/sdd1 -f /dev/sdc1 -r /dev/sdc1

Will Linux first mark sdc1 as faulty, and never read from it again, and then sync up sdd1 from sda1 and sdb1 (the two other disks in the array)?

If so then I'm vulnerable to the case that there is an unreadable block (even a single one!) on sda1 or sdb1, and that will cause the rebuild to fail.

What I want to do is have sdd1 be synced as a copy of sdc1 before marking sdc1 as faulty. Then I won't be in a situation where I don't have redundancy (albeit with one redundancy-stripe being on a disk that very well can give read errors).

Is there a way to do this online? Offline I can:

  • down the array (mdadm --stop)
  • dd sdc1 over to sdd1 (dd if=/dev/sdc1 of=/dev/sdd1)
  • take out sdc physically
  • bring the array up using the two old working ones and the new one (mdadm -A -s)
  • resync

Well, the problem with that method is that in the last step if there is a mismatch I want the new disk to be the one that is rewritten, not the parity (whatever disk that is on for that stripe).

So "rebuild sdd1 as a new sdc1, getting data from sda1 and sdb1, but if they fail, copy what's on sdc1".

4
  • This doesn't sound like something that should be community wiki.
    – Kevin M
    Jul 6, 2009 at 21:53
  • You're right. Silly of me.
    – Thomas
    Jul 6, 2009 at 22:02
  • Is there a way to remove the community wiki status?
    – Thomas
    Jul 6, 2009 at 22:04
  • no, see last chapter on: serverfault.com/questions/4035/…
    – ThorstenS
    Jul 7, 2009 at 2:43

4 Answers 4

5

Individually, those commands will not do what you desire.

mdadm /dev/md0 -a /dev/sdd1 
cat /proc/mdstat; #(you should now have a spare drive in raid5)
mdadm /dev/md0 -f /dev/sdc1
cat /proc/mdstat; #(you should now see a rebuild occuring to sdd1)

A test of the actual command does indeed cause a rebuild to occur.

Alas, I don't believe you can do what you desire right now.

As an aside, I often reference the linux raid wiki, and experiment on what I see there using loopback files.

dd if=/dev/zero of=loopbackfile.0 bs=1024k count=100
losetup /dev/loop0 loopbackfile.0

That gives you 100 meg file that is available as /dev/loop0. Create another couple of them, and you can use mdadm (e.g. "mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/loop0 /dev/loop1 /dev/loop2) without affecting real drives or data.


Note I formerly had said that

mdadm /dev/md0 -a /dev/sdd1
mdadm --grow /dev/md0 --raid-disks=4

would grow your array to a raid6. This is false. This will simply add a fourth disk to your array, which does not put you in any better of a position than you are currently in.


1
  • Is there any value added by doing the add and faulty in separate commands, as opposed to the one I used (pretty much from the manpage) of "mdadm /dev/md0 -a /dev/sdd1 -f /dev/sdc1 -r /dev/sdc1"?
    – Thomas
    Jul 7, 2009 at 6:40
5

Test software raid in a sandbox!

I would suggest you play arround within a sandbox.
As mdadm can work with image-files and not just with devicefiles like
i.e. /dev/sda or /dev/mapper/vg00/lv_home - why don`t you test your migration
within a second softwarerraid on your machine :?)

Linux OS

I'm doing this under debian/lenny and bash:

# cat /etc/debian_version && uname -r && bash --version
5.0.2
2.6.26-2-amd64
GNU bash, version 3.2.39(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2007 Free Software Foundation, Inc.

Step 1

As root create 4x128MB diskimages like this (you need 512 MB free diskspace on /)

sudo su 
mkdir -p ~/raidtest/{root,home} && cd ~/raidtest
for i in sd{a,b,c,d} ; do
  dd if=/dev/zero bs=128 count=1M of=$i
done

Lets see whats happened:

# ls -hon --time-style=+
total 512M
drwxr-xr-x 2 0 4,0K  home
drwxr-xr-x 2 0 4,0K  root
-rw-r--r-- 1 0 128M  sda
-rw-r--r-- 1 0 128M  sdb
-rw-r--r-- 1 0 128M  sdc
-rw-r--r-- 1 0 128M  sdd

Step 2

partitioning the files

I create 3 partitions (20MB, 40MB and 56MB) for swap,/ and /home on sda through an loop-device:

# losetup /dev/loop0 sda
# ! echo "n
p
1

+20M
t
fd
n
p
2

+40M
t
2
fd
n
p
3


t
3
fd
w" | fdisk /dev/loop0

Ok, look whats happened:

# fdisk -l /dev/loop0
    Disk /dev/loop0: 134 MB, 134217728 bytes
255 heads, 63 sectors/track, 16 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xe90aaf21

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1               1           3       24066   fd  Linux raid autodetect
/dev/loop0p2               4           9       48195   fd  Linux raid autodetect
/dev/loop0p3              10          16       56227+  fd  Linux raid autodetect

Copy this partitionscheme to loop{1,2,3} ^= sd{b,c,d}

# losetup /dev/loop1 sdb
# sfdisk -d /dev/loop0 | sfdisk /dev/loop1
# losetup /dev/loop2 sdc
# sfdisk -d /dev/loop0 | sfdisk /dev/loop2
# losetup /dev/loop3 sda
# sfdisk -d /dev/loop0 | sfdisk /dev/loop3

Optional: If you have installed parted, run partprobe on the devices to update the kernels table

# partprobe /dev/loop0
# partprobe /dev/loop1
# partprobe /dev/loop2
# partprobe /dev/loop3

Step 3

Use kpartx to created the per partition devices under /dev/mapper/

aptitude install kpartx dmsetup
# kpartx -av /dev/loop0
add map loop0p1 (254:3): 0 48132 linear /dev/loop0 63
add map loop0p2 (254:4): 0 96390 linear /dev/loop0 48195
add map loop0p3 (254:5): 0 112455 linear /dev/loop0 144585
# kpartx -av /dev/loop1
add map loop1p1 (254:6): 0 48132 linear /dev/loop1 63
add map loop1p2 (254:7): 0 96390 linear /dev/loop1 48195
add map loop1p3 (254:8): 0 112455 linear /dev/loop1 144585
# kpartx -av /dev/loop2
add map loop2p1 (254:9): 0 48132 linear /dev/loop2 63
add map loop2p2 (254:10): 0 96390 linear /dev/loop2 48195
add map loop2p3 (254:11): 0 112455 linear /dev/loop2 144585
# kpartx -av /dev/loop3
add map loop3p1 (254:12): 0 48132 linear /dev/loop3 63
add map loop3p2 (254:13): 0 96390 linear /dev/loop3 48195
add map loop3p3 (254:14): 0 112455 linear /dev/loop3 144585

Step 4

create your raid5 and watch the status
We are still root! On my workstation I don`t use raid, just LVM, so I have to load the kernel module and install the package mdadm.

# modprobe raid5
# aptitude install mdadm
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
unused devices: <none>

I use md{10,11,12} for this test. Watch out, that they are not used on your system (this would be abnormal)!
--force and -x 0 are used, because otherwise mdadm puts one partition als spare:

## the 20MB Partition
# mdadm --create --force -l 5 -n3 -x 0 /dev/md10 /dev/mapper/loop0p1 /dev/mapper/loop1p1 /dev/mapper/loop2p1
mdadm: array /dev/md10 started.
## the 40MB Partition
# mdadm --create --force -l 5 -n3 /dev/md11-x 0 /dev/mapper/loop0p2 /dev/mapper/loop1p2 /dev/mapper/loop2p2
mdadm: array /dev/md11 started.
## the 56MB Partition
# mdadm --create --force -l 5 -n3 /dev/md12-x 0 /dev/mapper/loop0p3 /dev/mapper/loop1p3 /dev/mapper/loop2p3
mdadm: array /dev/md12 started.

How it looks like now:

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md12 : active raid5 dm-11[2] dm-8[1] dm-5[0]
      112256 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md11 : active raid5 dm-10[2] dm-7[1] dm-4[0]
      96256 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

md10 : active raid5 dm-9[2] dm-6[1] dm-3[0]
      48000 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>

Info
the output isn't nice. mdstat shows just dm-3 .. dm-11, meaing /dev/mapper/loop*
but ls -lsa /dev/disk/by-id shows you the current mapping.

My output on md10 begins with dm-9 (meaning /dev/mapper/loop0p1), because of tests I did while writing this article and my LVM uses dm-{0,1,2}.
You can also use mdadm --examine --scan or more detailed infos via mdadm -Q --detail /dev/md10 /dev/md11 /dev/md12

Step 5

As root create silently filesystems and swap

# mkswap /dev/md10 > /dev/null 2>&1
# mke2fs -m0 -Lroot /dev/md11 -F > /dev/null 2>&1
# mke2fs -m0 -Lhome /dev/md12 -F > /dev/null 2>&1

Mount your new raiddevices:

# swapon /dev/md10
# mount /dev/md11 root/
# mount /dev/md12 home/

Have a look at the structure and if /dev/md10 is a valid swap-partition:
(my workstation also uses /dev/mapper/vg00-swap, therefore the higher priority)

# \tree
.
|-- home
|   `-- lost+found
|-- root
|   `-- lost+found
|-- sda
|-- sdb
|-- sdc
`-- sdd

# cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/mapper/vg00-swap                   partition       9764856 53688   -1
/dev/md10                               partition       47992   0       -2

wow, much work for the sandbox - but its woth it, when your wanna play with mdadm - use it!

Now you have a running raid5 and can test the migration
I think there are some excellent answers here - test them carefully on your system!

Last step

After finishing your tests, shut down your mds and delete your /dev/loop*

# mdadm --stop /dev/md10
# mdadm --stop /dev/md11
# mdadm --stop /dev/md12
# kpartx -dv /dev/loop0
# kpartx -dv /dev/loop1
# kpartx -dv /dev/loop2
# kpartx -dv /dev/loop3

bringing it up again after a reboot

sudo su
cd ~/raidtest
# connecting the files to /dev/loop*
losetup /dev/loop0 sda
losetup /dev/loop1 sdb
losetup /dev/loop2 sdc
losetup /dev/loop3 sdd

# access to the partions in /dev/loop*
kpartx -av /dev/loop0
kpartx -av /dev/loop1
kpartx -av /dev/loop2
kpartx -av /dev/loop3

# start the raid again
mdadm --assemble /dev/md10 /dev/mapper/loop0p1 /dev/mapper/loop1p1 /dev/mapper/loop2p1
mdadm --assemble /dev/md11 /dev/mapper/loop0p2 /dev/mapper/loop1p2 /dev/mapper/loop2p2
mdadm --assemble /dev/md12 /dev/mapper/loop0p3 /dev/mapper/loop1p3 /dev/mapper/loop2p3

# show active raids
cat /proc/mdstat

After testing: copy partitiontable to /dev/sdd

Your tests went fine?
Ok, then you have to copy the partition from /dev/sda to /dev/sdd as we did in the sandbox with our files:

sfdisk -d /dev/sda | sfdisk /dev/sdd

Now you can add /dev/sdd to your raid

Info
If this fails, becaus of different harddisk vendors/models, you have to play with -uS (sectors), -uB (blocks), -uC (cylinders) or -uM (megabytes) - consult man sfdisk!

Some of my real-life raidcombos where P-ATA <-> P-ATA but even SCSCI <-> P-ATA works fine, unless the new devices size is equal or bigger then other harddisks.
Softwareraid ist so much flexible!

Update your /etc/mdadm/mdadm.conf

If you have an /etc/mdadm/mdadm.conf please look into and update it! mdadm can help you displaying the correct syntax:

mdadm --detail --scan

Good luck!

0

At the risk of stating the obvious, isn't the next step to make a backup? Or preferably two.

My experience with RAID5 rebuilds is there's a significant chance of a failure for whatever reason.

If you're concerned about the state of sda1 or sdb1, boot off a recovery CD and check the drives offline.

3
  • This is my home setup. I don't have room/money to have a backup of everything. Of course the most important stuff will be backed up. But all the virtual lab machines I'm playing with would take quite a while to reinstall.
    – Thomas
    Jul 6, 2009 at 21:46
  • I understand backups can be a hassle, but RAID isn't a backup strategy! Doing your drive swap with no backup is a significant risk. There's no getting around that. I think your best shot is to check the two good drives, off-line, to make sure they are readable before you do the swap.
    – John McC
    Jul 6, 2009 at 22:39
  • RAID isn't my backup strategy. That's not what I said. Yes, I will test-read the drives beforehand.
    – Thomas
    Jul 6, 2009 at 22:42
0

I am not certain, but I believe it may be possible with Linux raid to switch from a RAID5 to a RAID6.

If that is possible, then you could add the spare device, and if possible switch it over to a RAID6 so the parity would get regenerated onto the new drive in addition to the old drive. Once the sync is complete pull the failed drive, and switch back to a RAID5.

I strongly suggest you make a backup if you don't have one. The question you need to decide on, is not if you can afford to make a backup, the question is, will you be able to afford the loss of your data, the loss of your time, or the thousands you might spend on a drive recovery service.

5
  • That sounds implausible. Are you sure you're not thinking of growing a raid5 from n disks to n+1? Because that you can do in Linux.
    – Thomas
    Jul 6, 2009 at 22:18
  • Why does it sound implausible? A RAID6 is almost the same as a RAID5, it just has 2 parity devices instead of one.
    – Zoredache
    Jul 6, 2009 at 22:21
  • Apparently, this conversion is on the TODO list, but not possible yet. neil.brown.name/blog/20050727143147
    – Zoredache
    Jul 6, 2009 at 22:25
  • Note that the comment states "within the next 6 months", and it was written quite a while ago. But yes, color me surprised that it was even in the (comments of) the TODO.
    – Thomas
    Jul 6, 2009 at 22:41
  • Oh, and while I can't afford a backup of the whole thing I do back up (and transfer off-site and offline) the stuff that I can't afford to lose. I can afford to lose the rest. It's just that losing it would suck. "Can't afford" is probably better states as "is not worth the money and time, considering the risk".
    – Thomas
    Jul 7, 2009 at 6:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .