I have a Linux software raid 10 setup consisting of 5 RAID 1s (Two drives per mirrored setup) and a RAID 0 across all 5 RAID 1 pairs. To test that none of the drives were going to fail quickly under load I used badblocks across the RAID 0 with a destructive read/write mode.
Badblocks command: badblocks -b 4096 -c 98304 -p 0 -w -s /dev/md13
One of the devices failed and instead of the badblocks program happily moving on it hung. If I run a sync command this also hangs. First I would assume this isn't standard behavior for a RAID 1 device. If one of the drives fails it should still be able to write to the virtual device that the two drives make up without a problem.
So I proceeded to force fail the drive and try to remove it. I can set the drive to faulty without any problem (However the IO operations are still hung). I cannot remove the device entirely from the raid it says it is busy. My assumption is that if I can kick it out of the raid entirely the IO will continue but that is just an assumption and I do think I am dealing with a bug of sorts.
What is going on here exactly? Am I in an unrecoverable spot due to a bug?
The system is running kernel 2.6.18 so it isn't exactly new but I would think given that software raid has been around for so long issues like these would not happen.
Any insight is greatly appreciated.
mdadm --detail /dev/md13
Version : 00.90.03 Creation Time : Thu Jan 21 14:21:57 2010 Raid Level : raid0 Array Size : 2441919360 (2328.80 GiB 2500.53 GB) Raid Devices : 5
Total Devices : 5 Preferred Minor : 13 Persistence : Superblock is persistent
Update Time : Thu Jan 21 14:21:57 2010 State : clean Active Devices : 5 Working Devices : 5
Failed Devices : 0 Spare Devices : 0
Chunk Size : 64K UUID : cfabfaee:06cf0cb2:22929c7b:7b037984 Events : 0.3 Number Major Minor RaidDevice State 0 9 7 0 active sync /dev/md7 1 9 8 1 active sync /dev/md8 2 9 9 2 active sync /dev/md9 3 9 10 3 active sync /dev/md10 4 9 11 4 active sync /dev/md11
The failing raid output:
/dev/md8: Version : 00.90.03 Creation Time : Thu Jan 21 14:20:47 2010 Raid Level : raid1 Array Size : 488383936 (465.76 GiB 500.11 GB) Device Size : 488383936 (465.76 GiB 500.11 GB) Raid Devices : 2
Total Devices : 2 Preferred Minor : 8 Persistence : Superblock is persistent
Update Time : Mon Jan 25 04:52:25 2010 State : active, degraded Active Devices : 1 Working Devices : 1
Failed Devices : 1 Spare Devices : 0
UUID : 2865aefa:ab6358d8:8f82caf4:1663e806 Events : 0.11 Number Major Minor RaidDevice State 0 65 17 0 active sync /dev/sdr1 1 8 209 1 faulty /dev/sdn1