I just discovered the same problem on two brand new and identical servers installed only about 9 months ago. I was unable to write to the disk on both of them, because the system had marked it read-only. Logs indicated there was some sort of disk error on both.
Note that I'm running KVM with several guests on each of these servers. The guests were all running fine, but the problem was in the KVM host. This probably doesn't matter, but maybe it pertains. Both systems have only two drives with software raid1 and LVM on top. Each KVM guest has its own LVM partition as well.
Both systems were showing a degraded RAID1 array when looking at
So I rebooted one of the systems, and it told me I needed to manually run
fsck. So I did so. It appeared to fix the problems and a reboot brought the system back up normally. The same process worked on the 2nd server as well.
Next I ran
mdadm --manage /dev/md0 --add /dev/sdb1 to add the failed drive back into the array. This worked fine on both servers. For the next hour or so, looking at
/proc/mdstat showed progress being made on the drives syncing. After about an hour, one system finished, and
/proc/mdstat showed everything working nicely with
However, on the other system, after about 1.5 hours, the system load skyrocketed and nothing was responsive. A few minutes later, everything came back. But looking at
/proc/mdstat now shows the following:
root@bond:/etc# cat /proc/mdstat Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] md0 : active raid1 sda1 sdb1 293033536 blocks [2/1] [_U] unused devices: <none>
As you can see, it appears to be no longer syncing. The percentage completed, time remaining, etc. is no longer show. However, running
mdadm --detail /dev/md0 shows this:
root@bond:/etc# mdadm --detail /dev/md0 /dev/md0: Version : 00.90 Creation Time : Mon Nov 30 20:04:44 2009 Raid Level : raid1 Array Size : 293033536 (279.46 GiB 300.07 GB) Used Dev Size : 293033536 (279.46 GiB 300.07 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Fri Sep 10 23:38:33 2010 State : clean, degraded Active Devices : 1 Working Devices : 2 Failed Devices : 0 Spare Devices : 1 UUID : 4fb7b768:16c7d5b3:2e7b5ffd:55e4b71d Events : 0.5104310 Number Major Minor RaidDevice State 2 8 1 0 spare rebuilding /dev/sda1 1 8 17 1 active sync /dev/sdb1
The bottom line seems to indicate that the spare is rebuilding. Why is it a spare? The system is reporting both devices as clean. It has stayed like this for hours. The drives are small and fast 300GB 10K RPM VelociRaptors, so I would think it would have sync'ed by now. Attempting to re-add says the device is busy:
root@bond:/etc# mdadm /dev/md0 --re-add /dev/sda mdadm: Cannot open /dev/sda: Device or resource busy
Running dmesg on the "good" server shows this at the end:
[ 4084.439822] md: md0: recovery done. [ 4084.487756] RAID1 conf printout: [ 4084.487759] --- wd:2 rd:2 [ 4084.487763] disk 0, wo:0, o:1, dev:sda1 [ 4084.487765] disk 1, wo:0, o:1, dev:sdb1
On the "bad" server, those last 4 lines are repeated hundreds of times. On the "good" server, they only show once.
Are the drives still syncing? Will this "rebuild" finish? Do I just need to be more patient? If not, what should I do now?
I just rebooted, and the drive started syncing all over again. After almost 2 hours, the same thing happened as described above (still get a [_U]). However, I was able to see the dmesg logs before the RAID1 conf printout chunks consumed it all:
[ 6348.303685] sd 1:0:0:0: [sdb] Unhandled sense code [ 6348.303688] sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 6348.303692] sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor] [ 6348.303697] Descriptor sense data with sense descriptors (in hex): [ 6348.303699] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 [ 6348.303707] 22 ee a4 c7 [ 6348.303711] sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed [ 6348.303716] end_request: I/O error, dev sdb, sector 586065095 [ 6348.303753] ata2: EH complete [ 6348.303776] raid1: sdb: unrecoverable I/O read error for block 586065024 [ 6348.305625] md: md0: recovery done.
So maybe the question I should be asking is "How do I run fsck on a spare disk in a raid set?"