Today I'm in a very bad place. A production raid array went down. The controller failed. After unmounting the file systems and killing all processes, I changed the hardware controller. On restarting the array the disks have come back and I want to try to re assemble the array.
However, I cannot stop the existing array.
I have :
cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md1 : active raid5 sdi1[5](F) sdh1[6](F) sdg1[7](F) sdf1[8](F) sde1[9](F) sdd1[10](F)
976751616 blocks level 5, 64k chunk, algorithm 2 [5/0] [_____]
md2 : active raid5 sdo1[5](F) sdm1[6](F) sdl1[7](F) sdk1[8](F) sdj1[9](F)
3907039744 blocks level 5, 64k chunk, algorithm 2 [5/0] [_____]
unused devices: <none>
The disks are now /dev/sdn sds sdt sdu sdv and sdw + sdx sdy sdz sdaa and sdab
Trying to stop the array gives:
[root@tango ~]# mdadm --stop /dev/md1
mdadm: fail to stop array /dev/md1: Device or resource busy
Perhaps a running process, mounted filesystem or active volume group?
This is because it thinks the volume groups are still active.
lvs gives:
/dev/md1: read failed after 0 of 4096 at 0: Input/output error
/dev/md2: read failed after 0 of 4096 at 0: Input/output error
/dev/raid5_vol2/u03: read failed after 0 of 4096 at 0: Input/output error
/dev/raid5_vol1/u06: read failed after 0 of 4096 at 0: Input/output error
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
LogVol00 VolGroup00 -wi-ao 111.00G
LogVol01 VolGroup00 -wi-ao 17.50G
u01 raid1_vol1 -wi-ao 128.00G
u04 raid1_vol2 -wi-ao 128.00G
u05 raid5_vol3 -wi-ao 521.00G
u02 raid5_vol4 -wi-ao 521.00G
u07 raid5_vol5 -wi-ao 248.00G
trying to set the lv as inactive fails with:
[root@tango ~]# lvchange -a n /dev/raid5_vol1
/dev/md1: read failed after 0 of 4096 at 0: Input/output error
/dev/md2: read failed after 0 of 4096 at 0: Input/output error
/dev/raid5_vol2/u03: read failed after 0 of 4096 at 0: Input/output error
/dev/raid5_vol1/u06: read failed after 0 of 4096 at 0: Input/output error
/dev/md1: read failed after 0 of 4096 at 0: Input/output error
/dev/md1: read failed after 0 of 4096 at 1000193589248: Input/output error
/dev/md1: read failed after 0 of 4096 at 1000193646592: Input/output error
/dev/md1: read failed after 0 of 4096 at 0: Input/output error
/dev/md1: read failed after 0 of 4096 at 4096: Input/output error
/dev/md1: read failed after 0 of 4096 at 0: Input/output error
/dev/md2: read failed after 0 of 4096 at 0: Input/output error
/dev/md2: read failed after 0 of 4096 at 4000808632320: Input/output error
/dev/md2: read failed after 0 of 4096 at 4000808689664: Input/output error
/dev/md2: read failed after 0 of 4096 at 0: Input/output error
/dev/md2: read failed after 0 of 4096 at 4096: Input/output error
/dev/md2: read failed after 0 of 4096 at 0: Input/output error
/dev/raid5_vol2/u03: read failed after 0 of 4096 at 3974992166912: Input/output error
/dev/raid5_vol2/u03: read failed after 0 of 4096 at 3974992224256: Input/output error
/dev/raid5_vol2/u03: read failed after 0 of 4096 at 0: Input/output error
/dev/raid5_vol2/u03: read failed after 0 of 4096 at 4096: Input/output error
/dev/raid5_vol2/u03: read failed after 0 of 4096 at 0: Input/output error
/dev/raid5_vol1/u06: read failed after 0 of 4096 at 991063638016: Input/output error
/dev/raid5_vol1/u06: read failed after 0 of 4096 at 991063695360: Input/output error
/dev/raid5_vol1/u06: read failed after 0 of 4096 at 0: Input/output error
/dev/raid5_vol1/u06: read failed after 0 of 4096 at 4096: Input/output error
/dev/raid5_vol1/u06: read failed after 0 of 4096 at 0: Input/output error
Volume group "raid5_vol1" not found
Skipping volume group raid5_vol1
I also tried :
[root@tango ~]# mdadm --assemble --force /dev/md1 /dev/sdn /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw
mdadm: device /dev/md1 already active - cannot assemble it
So I'm out of ideas! Can anyone help?
Katie
[root@tango ~]# cat /etc/mdadm.conf
DEVICE partitions
ARRAY /dev/md2 level=raid5 num-devices=5 metadata=0.90 spares=1 UUID=dce41d37:804dea59:e5727503:733c194b
ARRAY /dev/md1 level=raid5 num-devices=5 metadata=0.90 spares=2 UUID=9e8a7463:ea5942af:abf7ab25:de5bb614
pvscan; vgreduce --removemissing raid5_vol1
(from docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/…)/etc/mdadm/mdadm.conf
? Presumably it identifies the arrays by UUID; can you just reboot the machine? With 2 dead arrays it's probably not doing anything useful. Alternatively, have you tried removing one of the 'failed' drives (mdadm /dev/md1 --remove /dev/sdi1
for example) and then adding its new version (if you can identify which one it is). If that works, then you can repeat until they are all back.