3

Today I'm in a very bad place. A production raid array went down. The controller failed. After unmounting the file systems and killing all processes, I changed the hardware controller. On restarting the array the disks have come back and I want to try to re assemble the array.

However, I cannot stop the existing array.

I have :

cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]
md1 : active raid5 sdi1[5](F) sdh1[6](F) sdg1[7](F) sdf1[8](F) sde1[9](F) sdd1[10](F)
      976751616 blocks level 5, 64k chunk, algorithm 2 [5/0] [_____]

md2 : active raid5 sdo1[5](F) sdm1[6](F) sdl1[7](F) sdk1[8](F) sdj1[9](F)
      3907039744 blocks level 5, 64k chunk, algorithm 2 [5/0] [_____]

unused devices: <none>

The disks are now /dev/sdn sds sdt sdu sdv and sdw + sdx sdy sdz sdaa and sdab

Trying to stop the array gives:

[root@tango ~]# mdadm --stop /dev/md1
mdadm: fail to stop array /dev/md1: Device or resource busy
Perhaps a running process, mounted filesystem or active volume group?

This is because it thinks the volume groups are still active.

lvs gives:

  /dev/md1: read failed after 0 of 4096 at 0: Input/output error
  /dev/md2: read failed after 0 of 4096 at 0: Input/output error
  /dev/raid5_vol2/u03: read failed after 0 of 4096 at 0: Input/output error
  /dev/raid5_vol1/u06: read failed after 0 of 4096 at 0: Input/output error
  LV       VG         Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  LogVol00 VolGroup00 -wi-ao 111.00G
  LogVol01 VolGroup00 -wi-ao  17.50G
  u01      raid1_vol1 -wi-ao 128.00G
  u04      raid1_vol2 -wi-ao 128.00G
  u05      raid5_vol3 -wi-ao 521.00G
  u02      raid5_vol4 -wi-ao 521.00G
  u07      raid5_vol5 -wi-ao 248.00G

trying to set the lv as inactive fails with:

[root@tango ~]# lvchange -a n /dev/raid5_vol1
  /dev/md1: read failed after 0 of 4096 at 0: Input/output error
  /dev/md2: read failed after 0 of 4096 at 0: Input/output error
  /dev/raid5_vol2/u03: read failed after 0 of 4096 at 0: Input/output error
  /dev/raid5_vol1/u06: read failed after 0 of 4096 at 0: Input/output error
  /dev/md1: read failed after 0 of 4096 at 0: Input/output error
  /dev/md1: read failed after 0 of 4096 at 1000193589248: Input/output error
  /dev/md1: read failed after 0 of 4096 at 1000193646592: Input/output error
  /dev/md1: read failed after 0 of 4096 at 0: Input/output error
  /dev/md1: read failed after 0 of 4096 at 4096: Input/output error
  /dev/md1: read failed after 0 of 4096 at 0: Input/output error
  /dev/md2: read failed after 0 of 4096 at 0: Input/output error
  /dev/md2: read failed after 0 of 4096 at 4000808632320: Input/output error
  /dev/md2: read failed after 0 of 4096 at 4000808689664: Input/output error
  /dev/md2: read failed after 0 of 4096 at 0: Input/output error
  /dev/md2: read failed after 0 of 4096 at 4096: Input/output error
  /dev/md2: read failed after 0 of 4096 at 0: Input/output error
  /dev/raid5_vol2/u03: read failed after 0 of 4096 at 3974992166912: Input/output error
  /dev/raid5_vol2/u03: read failed after 0 of 4096 at 3974992224256: Input/output error
  /dev/raid5_vol2/u03: read failed after 0 of 4096 at 0: Input/output error
  /dev/raid5_vol2/u03: read failed after 0 of 4096 at 4096: Input/output error
  /dev/raid5_vol2/u03: read failed after 0 of 4096 at 0: Input/output error
  /dev/raid5_vol1/u06: read failed after 0 of 4096 at 991063638016: Input/output error
  /dev/raid5_vol1/u06: read failed after 0 of 4096 at 991063695360: Input/output error
  /dev/raid5_vol1/u06: read failed after 0 of 4096 at 0: Input/output error
  /dev/raid5_vol1/u06: read failed after 0 of 4096 at 4096: Input/output error
  /dev/raid5_vol1/u06: read failed after 0 of 4096 at 0: Input/output error
  Volume group "raid5_vol1" not found
  Skipping volume group raid5_vol1

I also tried :

[root@tango ~]# mdadm --assemble --force /dev/md1 /dev/sdn /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw
mdadm: device /dev/md1 already active - cannot assemble it

So I'm out of ideas! Can anyone help?

Katie


[root@tango ~]# cat /etc/mdadm.conf
DEVICE partitions
ARRAY /dev/md2 level=raid5 num-devices=5 metadata=0.90 spares=1 UUID=dce41d37:804dea59:e5727503:733c194b
ARRAY /dev/md1 level=raid5 num-devices=5 metadata=0.90 spares=2 UUID=9e8a7463:ea5942af:abf7ab25:de5bb614
9
  • What is your question exactly?
    – poige
    Aug 9, 2012 at 12:02
  • I want to get my array working again? How do I do that?
    – user131012
    Aug 9, 2012 at 12:04
  • In common words: Engineer it back to life.
    – poige
    Aug 9, 2012 at 12:08
  • Try . pvscan; vgreduce --removemissing raid5_vol1 (from docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/…)
    – phemmer
    Aug 9, 2012 at 12:21
  • What's the contents of /etc/mdadm/mdadm.conf? Presumably it identifies the arrays by UUID; can you just reboot the machine? With 2 dead arrays it's probably not doing anything useful. Alternatively, have you tried removing one of the 'failed' drives (mdadm /dev/md1 --remove /dev/sdi1 for example) and then adding its new version (if you can identify which one it is). If that works, then you can repeat until they are all back.
    – StarNamer
    Aug 9, 2012 at 13:23

1 Answer 1

2

I finally resolved this, 24 hours of pain so I thought I would share:

Here's the info, if you loose a disk controller then all your disks go into failed state. When you shutdown the array and replace the controller on restart all the disks have different labels. Notice I did not say reboot. I restarted the raid array as the system could not be rebooted.

Step 1 - Tell mdadm to forget about detached disk like this:

$>mdadm /dev/md1 --remove detached

If you try to use the mdadm /dev/md1 --remove /dev/sdX this will fail so you have to use 'detached'. If this does not detach all the disks in your raid group you will likely need to run it more than once. I did. On the third attempy cat /proc/mdstat showed them all as removed.

Step 2 - If you try to shutdown the array and you had any active logical volumes or users at the time the controller failed you will not be able to:

$>mdadm --stop /dev/md1

However if this works ignore step 3.

Step 3 - using dmsetup remove to free up the array

So.. you need to run the dmsetup remove -f -j 1234 -m 1 command. These numbers come from running dmsetup info and looking for the volumes that have failed. In my case they were easy to identify because the broken ones had an open flag of zero like this. Running this on a live system is scarery so make sure you get these numbers right.This command will help:

$>dmsetup info .. .. .. Name: raid5_vol2-u03 State: ACTIVE Read Ahead: 1024 Tables present: LIVE Open count: 0 Event number: 0 Major, minor: 253, 6 Number of targets: 1 UUID: LVM-fPJWYWhBhTO7tzHMBbc01qR20Nazc8N93S0bMPqstrg0sFFVtipaWYi7NYzas12d

Volume on the box that were active had a 1 in Open Count. The broken ones had an open count of 0.

If you are sure you have the right volume then you can issue the command. In my case the two numbers were 253 and 6.

$>dmsetup remove -f -j 253 -m 6 wait 30 seconds $>mdadm --stop /dev/md1

Step 4 - assumine you have been able to stop the array you need to try and reassemble it.

To do this you need to know what the new disk labels are, if you know the serial numbers of your disks before the crash which is good practice you can run

$>smartctl -i /dev/sdN to find out which label matched which disk now. Alternatively you could shutdown the array again and physically check them.

Once you know:

$>mdadm --assemble /dev/md1 /dev/sdn1 /dev/sds1 /dev/sdt1 ... etc.

If this works you will get a raid started messaged if not then hmm... Im not sure!

wait a bit ...

$>cat /proc/mdstat

The array should be syncing/reconstructing. My small one took minutes the big one took hours. If that works.

Step 5 - at this point the volume groups, logical volumes etc are all going to be in a state to fix them I ran:

$>pvscan $>vgscan $>lvscan

Luckily for me this identified my array and wired things up, but left the volume group as inactive.

So lvs showed:

inactive raid5_vol1 ...

Step 6 - make the volume group active again:

vgchange -a y raid5_vol1

Step 7 - If all went well this would not be mountable again and you can thank me in this post for helping.

Good luck!

Katie

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .