Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I just discovered the same problem on two brand new and identical servers installed only about 9 months ago. I was unable to write to the disk on both of them, because the system had marked it read-only. Logs indicated there was some sort of disk error on both.

Note that I'm running KVM with several guests on each of these servers. The guests were all running fine, but the problem was in the KVM host. This probably doesn't matter, but maybe it pertains. Both systems have only two drives with software raid1 and LVM on top. Each KVM guest has its own LVM partition as well.

Both systems were showing a degraded RAID1 array when looking at /proc/mdstat.

So I rebooted one of the systems, and it told me I needed to manually run fsck. So I did so. It appeared to fix the problems and a reboot brought the system back up normally. The same process worked on the 2nd server as well.

Next I ran mdadm --manage /dev/md0 --add /dev/sdb1 to add the failed drive back into the array. This worked fine on both servers. For the next hour or so, looking at /proc/mdstat showed progress being made on the drives syncing. After about an hour, one system finished, and /proc/mdstat showed everything working nicely with [UU].

However, on the other system, after about 1.5 hours, the system load skyrocketed and nothing was responsive. A few minutes later, everything came back. But looking at /proc/mdstat now shows the following:

root@bond:/etc# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid1 sda1[2] sdb1[1]
      293033536 blocks [2/1] [_U]

unused devices: <none>

As you can see, it appears to be no longer syncing. The percentage completed, time remaining, etc. is no longer show. However, running mdadm --detail /dev/md0 shows this:

root@bond:/etc# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90
  Creation Time : Mon Nov 30 20:04:44 2009
     Raid Level : raid1
     Array Size : 293033536 (279.46 GiB 300.07 GB)
  Used Dev Size : 293033536 (279.46 GiB 300.07 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Sep 10 23:38:33 2010
          State : clean, degraded
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

           UUID : 4fb7b768:16c7d5b3:2e7b5ffd:55e4b71d
         Events : 0.5104310

    Number   Major   Minor   RaidDevice State
       2       8        1        0      spare rebuilding   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1

The bottom line seems to indicate that the spare is rebuilding. Why is it a spare? The system is reporting both devices as clean. It has stayed like this for hours. The drives are small and fast 300GB 10K RPM VelociRaptors, so I would think it would have sync'ed by now. Attempting to re-add says the device is busy:

root@bond:/etc# mdadm /dev/md0 --re-add /dev/sda
mdadm: Cannot open /dev/sda: Device or resource busy

Running dmesg on the "good" server shows this at the end:

[ 4084.439822] md: md0: recovery done.
[ 4084.487756] RAID1 conf printout:
[ 4084.487759]  --- wd:2 rd:2
[ 4084.487763]  disk 0, wo:0, o:1, dev:sda1
[ 4084.487765]  disk 1, wo:0, o:1, dev:sdb1

On the "bad" server, those last 4 lines are repeated hundreds of times. On the "good" server, they only show once.

Are the drives still syncing? Will this "rebuild" finish? Do I just need to be more patient? If not, what should I do now?

UPDATE:

I just rebooted, and the drive started syncing all over again. After almost 2 hours, the same thing happened as described above (still get a [_U]). However, I was able to see the dmesg logs before the RAID1 conf printout chunks consumed it all:

[ 6348.303685] sd 1:0:0:0: [sdb] Unhandled sense code
[ 6348.303688] sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 6348.303692] sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]
[ 6348.303697] Descriptor sense data with sense descriptors (in hex):
[ 6348.303699]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 
[ 6348.303707]         22 ee a4 c7 
[ 6348.303711] sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
[ 6348.303716] end_request: I/O error, dev sdb, sector 586065095
[ 6348.303753] ata2: EH complete
[ 6348.303776] raid1: sdb: unrecoverable I/O read error for block 586065024
[ 6348.305625] md: md0: recovery done.

So maybe the question I should be asking is "How do I run fsck on a spare disk in a raid set?"

share|improve this question

3 Answers 3

I'm unclear on whether you actually replaced the failed drive(s)? Because your symptoms would make sense to me if you'd re-added the faulty drive, in which case there's a good chance the drive has locked up. If you did re-add the faulty drive, are there subsequent errors in /var/log/messages or dmesg?

(Incidentally, I'd strongly recommend against ever re-addeding a faulty drive to a RAID array. If the fault corrupted data on the platter you may find that when you add it back to the array, the resync leaves the corrupted file on the disc, and next time you read the files, it'll be a crapshoot as to whether you get good or bad data, depending on which disk responds first; I have seen this happen in the wild.)

share|improve this answer
    
To reply to your update: that's a fatal hardware failure on sdb. Pull it and replace it. fsck won't do anything for it. –  Rodger Sep 12 '10 at 7:49
    
thanks, will do. How can you tell this is a fatal error? I plan to run the WD diags on it to confirm this once I can schedule some downtime. –  Tauren Sep 12 '10 at 13:21
    
"raid1: sdb: unrecoverable I/O read error for block 586065024" - It couldn't correct the problem by remapping to a different block, so you've lost a block on the disk. This would tend to indicate the disk is on the way out. You could followup with smartmon tools to see if the SMART data for the drive confirms that. –  Rodger Sep 18 '10 at 9:02

Using mdadm --details will list a drive as spare while it's rebuilding. After rebuild is complete it will no longer show as spare.

[ 6348.303711] sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
[ 6348.303716] end_request: I/O error, dev sdb, sector 586065095
[ 6348.303753] ata2: EH complete
[ 6348.303776] raid1: sdb: unrecoverable I/O read error for block 586065024
[ 6348.305625] md: md0: recovery done.

The first line, states there was reallocation failure, and the data was not read. The following three lines are pointing out that the data could not be read, and lists the sectors that are unreadable.

As Rodger pointed out, the drive is bad, don't re-add it. It's never a good idea to re-add a drive that failed. Pull the drive and replace it. If you wish, run diagnostics on the failed drive but only after it's been pulled and replaced.

share|improve this answer

First- yes get rid of any disk that is throwing read errors that end up in the log file. This means the bad block relocation has failed and/or the drive is close to dying.

I suggest to rescue your data you use a Linux rescue CD such as http://ubuntu-rescue-remix.org/ to use ddrescue. This can do an image copy to a new disk's partition, and will do lots of retries etc to try to recover your partition. Mount a USB drive or another partition

mkdir /tmp/x && mount /dev/sdd1 /tmp/x

to hold the ddrescue log file- then you can stop the ddrescue (ctrl-C) and restart it later from the same point.

Make a partition on the new disk a bit bigger than the old disk. You don't have to use the whole disk!

Boot the rescue CD with "nodmraid" as a kernel boot parameter. If using ubuntu live CD then install RAID and LVM if you're using it

apt-get install mdadm lvm2 gddrescue

you will need to be on the internet for this to work). Otherwise use the ubuntu rescue CD for the ddrescue step. I swapped between the rescue CD for ddrescue runs, and the live CD for the grub and fsck work.

Assuming /dev/sdb is your failing source disk, and /dev/sdx is your new disk, and /mnt/x is a USB key or a partition on another disk that has been mounted. You need the ddrescue log file, really! As it tracks how ddrescue is going, and allows it to be interrupted.

As per http://www.forensicswiki.org/wiki/Ddrescue

ddrescue --no-split /dev/sdb /dev/sdX imagefile /mnt/x/logfile

then

ddrescue --direct --max-retries=3 /dev/sdb /dev/sdX /mnt/x/logfile

then

ddrescue --direct --retrim --max-retries=3 /dev/sdb /dev/sdX /mnt/x/logfile

Don't be afraid to Ctrl-C the process if it is taking hours to recover a single sector. Just go on to the next step (step 1 should succeed no matter what). The last step tries to recover the last crumbs of usable data.

You will also have to do

mdadm --create /dev/md99 --level-1 --raid-devices=2 missing /dev/sdX

to make a new RAID array using the new disk, this writes a new RAID superblock on the partition (in the last 64K to 128K at the end of the partition).

Remove your old failing disk /dev/sdb from the system so it is not visible to linux.

Make your source RAID disk accessible. You may have to use the "nodmraid" parameter to the kernel booting kernel, as I had problems with the ubuntu rescue CD, and ended up using the Ubuntu live CD (10.4) where nodmraid is in F6 Options. You should just need to use

mdadm --assemble /dev/md99 /dev/sdX

Then fsck or do whatever check you need to do to the data on the md99 RAID array (I used vgscan then was able to see the LVM LVs to run the check against). I use XFS for mythtv but xfs_check command crashed my system, but xfs_repair was OK.

Mount the /boot directory from your new /dev/sdX

mount /dev/mapper/my_vg/root_lv /tmp/x

then put a new GRUB boot record on the new /dev/sdX RAID disk (only if you boot from RAID!)

grub-setup -d /tmp/x/boot/grub /dev/sdX

now you have an (almost) bootable RAID array. You may also do the setup using GRUB itself, or use dd to copy the first 446 bytes of /dev/sdb to /dev/sdX. ONLY the first 446 bytes, the rest of the 1st sector is your partition table, which you will stuff up mightily if you copy more! You may also have to do the same for the 1st sector in your partition /dev/sdX1 (say). Back up any sectors you are going to overwrite, also using dd.

If using grub2 and you're booting from RAID, you will find that the RAID array UUID has changed so your boot will fail. Edit the boot command line (e in Grub startup panel) to remove splash and quiet, so you can see what is happening. Then after the failed boot you are left in initramfs.

mdadm --assemble /dev/md99 /dev/sdX

then check /proc/mdstat to make sure the array is there. If it is then just "exit" and hopefully your GRUB boot stanza will work OK (mine was set to use LVM so it just found the LVs on the RAID device once there was any RAID device there, it just searched for the LV). Once you are booted up then you are almost done.

The initrd image file (gzipped cpio file) contains a copy of mdadm.conf used during the boot process, visible and editable as /etc/mdadm/mdamdm.conf during the boot process. If you can get your system booted normally just update the initramfs using

update-initramfs -u

If you can't get the system booted because of the mismatched UUID in the mdadm.conf file

Be aware that your destination device /dev/sdX may appear as /dev/sdY when you boot a different way (Grub, rescue, real boot).

By the way, unless you are using RAID5 and are really interested in block alignment, I'd use a partition for your RAID array, you don't have to use a whole disk (esp if you are replacing a 1TB disk with a 2TB one). You can always add another partition and a 2nd RAID array later to use up the whole 2TB.

Phew! Done!

share|improve this answer
2  
-1 for giving bad advice on the raid-1. There is no need to try to save a failed HDD, and Raid1 is designed to survive down to 1 disk. Just get a new disk. Add it in to the Raid array. If you want to make sure you can boot from that disk then install grub on it as well. I would recomend that you install grub on both of the disks if they are in a raid1, that way you can easily recover if one drive dies and you have to power off to replace the disk. –  Squidly Oct 29 '11 at 0:14

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.