I have 4 disks RAID-5ed to create md0, and another 4 disks RAID-5ed to create md1. These are then combined via LVM to create one partition.

There was a power outage while I was gone, and when I got back, it looked like one of the disks in md1 was out of sync - mdadm kept claiming that it only could find 3 of the 4 drives. The only thing I could do to get anything to happen was to use mdadm --create on those four disks, then let it rebuild the array. This seemed like a bad idea to me, but none of the stuff I had was critical (although it'd take a while to get it all back), and a thread somewhere claimed that this would fix things. If this trashed all of my data, then I suppose you can stop reading and just tell me that.

After waiting four hours for the array to rebuild, md1 looked fine (I guess), but the lvm was complaining about not being able to find a device with the correct UUID, presumably because md1 changed UUIDs. I used the pvcreate and vgcfgrestore commands as documented here. Attempting to run an lvchange -a y on it, however, gives me a resume ioctl failed message.

Is there any hope for me to recover my data, or have I completely mucked it up?

My LV was set up linearly. It seems that I should be able to recover at least the data from md0 by using something like TestDisk. However, examples show LVM partitions listed in TestDisk, while I only see sda,sdb,etc. and md0 and md1.

[$]> lvdisplay
  --- Logical volume ---
  LV Name                /dev/VolGroup/lvm
  VG Name                VolGroup
  LV UUID                b1h1VL-f11t-TGJv-ajbx-4Axn-wi89-fReUKu
  LV Write Access        read/write
  LV Status              suspended
  # open                 0
  LV Size                6.14 TiB
  Current LE             1609648
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           254:0

[$]> lvchange -ay /dev/VolGroup/lvm
  device-mapper: reload ioctl failed: Invalid argument
  • Very detailed question. Wish I knew the answer. – hobodave Jan 3 '11 at 9:07
  • BTW, RAID5 should be avoided for larger disks, you should be looking at RAID6, or RAID10 these days. zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162 – Zoredache Jan 3 '11 at 19:43
  • All that says is what we already knew - RAID5 only has one spare drive, while RAID6 has two. For financial and logical reasons, my RAID sets were destined to be comprised of 4 disks, and RAID6 would've dropped my storage capacity below what I consider acceptable. FWIW, it wouldn't have helped me here, either, since the problem was all user-related >_>. – Xiong Chiamiov Jan 3 '11 at 20:02

I'm afraid that, by combining two (RAID) disks into a LVM volume and then deleting and recreating one of those disks, you actually have lost your data. If you used LVM striping, they're definitely gone, since every other chunk of your files was on that disk. With linear mode, you may be able to regain a part of your files with some disk recovery software.

Pity, because your basic configuration is sensible. Because of RAID5, you should have been able to access the volume even with one disk missing. I'd have suggested a backup before the "mdadm --create"...

  • Pooey, but about what I expected; I can't say what prompted me to go mucking around without knowing what I was doing (guess it's that male instinct to avoid asking for help). For the future, can you outline the steps that I should have taken in this situation? – Xiong Chiamiov Jan 3 '11 at 18:01
  • @Xiong Chiamiov, to deal with that situation in the future, you should have, a backup system in place. – Zoredache Jan 3 '11 at 19:41
  • Well, sure, backups are always good, but I mean, I choose RAID5 for redundancy's sake - there should be a way for me to fix one of the disks in the array without mucking up my LVM. Otherwise, what's the point? – Xiong Chiamiov Jan 3 '11 at 19:58

Next time, you've two options ... either you can manually attempt to assemble the raid using mdadm --assemble instead of --create. Or you can use "ckraid" similarly to the filesystem check, but for raid 5 arrays. ( The Linux Software Raid How-To mentions the latter ).

Personally, though, if you're using SW Raid without a UPS I'd personally go for Raid 10 instead of 5 or 6, since it should be more tolerant of power issues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.