6

To make a long story short, I am responsible for a media server that has two RAID arrays (/dev/md0 and /dev/md2). I still have a ton to learn about software RAID and LVM, etc. We had a drive fail in /dev/md0, okay not a big deal I replaced the bad drive, reassembled the RAID array (which took most of last night to complete) and this morning I came in and was able to mount /dev/md0 with no data loss.

Little did I know, something became corrupt with /dev/md2. Now /dev/md0 was set-up by my co-worker two years ago, I set-up /dev/md2 about six months ago. Always having done RAID using the hardware controllers built into servers from Dell and HP, this is the first time I have had to deal with software RAID. I followed the instructions on [url]http://www.gagme.com/greg/linux/raid-lvm.php[/url] which I felt was the best idea as LVM would give me the ability to grow the RAID array in the future (I just now learned that mdadm can do the same). There was no issues with this until today. After rebooting the machine after it repaired /dev/md0 it was unable to mount /dev/lmv-raid/lvm0 (said the file or directory couldn't be found). It almost looks like the logical volume is gone. I do not understand what would have caused this. No drives in the second array were bad, I didn't touch the second array. Nothing should have gone bad with the array. But it looks like something isn't right.

Now /dev/md2 is running and is in a clean state, but I cannot mount it. I am afraid that data is gone. The really terrible thing is that there is no explanation in my mind as to what happened. I am assuming that I can't mount /dev/md2 because the RAID array is expecting to be controlled by LVM. Maybe I do not understand the system well enough, but if I have a clean array shouldn't the data still be in the array? Any idea how I can go about recovering the data? The last thing I want to do is have to go back to my boss and tell him that in the process of fixing one array another array magically broke and all your data is gone. I have looked at multiple sites, but since I don't know what actually broke with it all I don't know if I have really lost data or if there is something else that I am missing.

Any advice would be greatly appreciated, I have been struggling with this all morning long and am at a total loss.

Here is some more information:

 [root@viamao ~]# mdadm --detail /dev/md2
 /dev/md2:
 Version : 00.90.01
 Creation Time : Tue Jun  9 11:32:06 2009
 Raid Level : raid5
 Array Size : 4395415488 (4191.79 GiB 4500.91 GB)
 Device Size : 1465138496 (1397.26 GiB 1500.30 GB)
 Raid Devices : 4
 Total Devices : 4
 Preferred Minor : 2
 Persistence : Superblock is persistent

Update Time : Tue Jun  9 15:54:39 2009
      State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

     Layout : left-symmetric
 Chunk Size : 64K

Number   Major   Minor   RaidDevice State
   0       8      128        0      active sync   /dev/sdi
   1       8      144        1      active sync   /dev/sdj
   2       8      160        2      active sync   /dev/sdk
   3       8      176        3      active sync   /dev/sdl
       UUID : 88bbe155:b6d852b1:3ad8a77d:5eba3ee2
     Events : 0.36


  [root@viamao archive2]# mount -t ext3 /dev/md2 /share/archive2
  mount: wrong fs type, bad option, bad superblock on /dev/md2,
  or too many mounted file systems


 [root@viamao archive2]# mount
 /dev/hda3 on / type ext3 (rw)
 none on /proc type proc (rw)
 none on /sys type sysfs (rw)
 none on /dev/pts type devpts (rw,gid=5,mode=620)
 usbfs on /proc/bus/usb type usbfs (rw)
 /dev/hda1 on /boot type ext3 (rw)
 none on /dev/shm type tmpfs (rw)
 none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
 sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
 /dev/md0 on /share/archive1 type ext3 (rw)


  [root@viamao share]# cat /proc/mdstat
  Personalities : [raid5] 
  md2 : active raid5 sdl[3] sdk[2] sdj[1] sdi[0]
  4395415488 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

  md0 : active raid5 sda[0] sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1]
  5128021248 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

  unused devices: <none>

Anything else I can show you that might be able to help? I have a active RAID array that I can't mount, thinking this has something to do with the fact that this used to be on lvm, but now when I try to mount /dev/lvm-raid/lvm0 it tells me the special device cannot be found. lvdisplay doesn't give me anything pvdisplay only shows me the drive that my OS is on. vgdisplay also yields nothing.

Thoughts? I am seriously at a loss here.

5 Answers 5

4

Filesystem layers in Linux (starting in reverse order, from the physical drive to the filesystem):

  1. physical device
    • /dev/sdi
    • /dev/sdj
    • /dev/sdk
    • /dev/sdl
  2. special md partition type on each drive (when used)

    • this may or may not be present. While it is recommended that you simply make single partitions on each drive that spans the entire size of the drive it is on, it is possible to specify the entire drive by using the device name directly. Note that this can cause some partition tools to become confused about what they are dealing with (because the partition table simply "goes away"), so I don't recommend it.

    In your case, the entire drives are specified, so there are no partitions to see. You won't have to worry about this.

  3. md driver (when used)

    • /dev/md2

    Your output from both the detail and /proc report that the array is up on all drives and no drives are in a failed state. This means the array is healthy!

  4. LVM (when used)

    • Type the following into a shell while logged in as root:

    pvscan && vgscan && lvscan

    If there are any volumes to be found, they should be here. Note that the volume scan process is controlled by a file that can choose certain devices to ignore when it performs the scan. You'll want to make sure that you scan /dev/md2 explicitly. Each LVM volume has a GUID imprinted into it; if this is lost or corrupted, it can cause some of the issues you are seeing. The goal here is to get your LVM volumes recognized. Once they are healthy, you'll be in good shape.

  5. filesystem

    I think you know the drill here.

From here you should be able to recover your filesystem(s).

3

usually LVM is done 'on top of' MD. maybe you setup LVM using command line and not your distro's tools? if so, maybe the startup scripts don't know about the LVM.

first do a 'vgscan' and see if it comes up. if so, it's just an issue of untangling the scripts.

6
  • Nope, vgscan doesn't show it. I had set-up the LVM in the command line and it was working till recently.
    – sxanness
    Jun 9, 2009 at 20:19
  • try pvscan and then check if the PV is there (pvs or pvdisplay).
    – Javier
    Jun 9, 2009 at 20:29
  • [root@viamao ~]# pvscan Incorrect metadata area header checksum Incorrect metadata area header checksum Incorrect metadata area header checksum PV /dev/hda2 lvm2 [37.17 GB] Total: 1 [37.17 GB] / in use: 0 [0 ] / in no VG: 1 [37.17 GB] /dev/hda2 is where the OS is, it is also reading the other partitions on the system that do not have LVM support.
    – sxanness
    Jun 9, 2009 at 20:33
  • [root@viamao ~]# pvdisplay Incorrect metadata area header checksum Incorrect metadata area header checksum Incorrect metadata area header checksum Incorrect metadata area header checksum --- NEW Physical volume --- PV Name /dev/hda2 VG Name PV Size 37.17 GB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID xAn1z6-1P0S-fN7e-bVUn-n65C-UNK0-2agMUx
    – sxanness
    Jun 9, 2009 at 20:34
  • can't untangle this. better add to your question.
    – Javier
    Jun 9, 2009 at 20:56
2

You may be challenged to get a "do this to fix it answer" on this largely because any good sysadmin is uber paranoid about data loss, including guiding someone else in a situation that may lead to data loss.

From what you've provided I'll summarize what I see an maybe where you can start.

  • /dev/md2 is a RAID 5 device with (4) 1.5TB drives
  • The entire underlying drive is being used by the raid module - no partitions on the drive
  • Your /dev/md2 device is now reporting a normal / happy status

Start by posting the results of: pvdisplay and vgscan

Do you have a "lvm-raid" file located in /etc/lvm/backup/ ?

2
  • Thank you Jeff for your response. And yes, my big worry right now here is that if the LVM stuff somehow got wiped out or become corrupt that I have just lost an entire RAID array's worth of data. Interesting enough in the /etc/lvm/backup there is a file called archive1 with a date of Feb 25, now this might be Feb 25 of 2008 as archive1 is no longer part of LVM (archive1 is /dev/md0) pvdisplay shows /dev/hda2 (part of OS) and vgscan shows nothing other than "incorrect metadata area checksum" due the partitions not part of lvm. I think I am pretty much screwed.
    – sxanness
    Jun 9, 2009 at 21:05
  • you can use vgcfgrestore to write back that config to the disks, but that should be a last resort. first be sure that your startup scripts and lvm.conf are correct.
    – Javier
    Jun 9, 2009 at 21:32
2

The first thing I would do in this kind of situation, if at all possible: Make an exact copy of both disks forming /dev/md2 (with dd or something like that). This may takes ages, but if you frack up things even more while attempting to repair this, you can go back to where you started.

1

You might try the following

pvdisplay /dev/md2

and provide the results. This will explicitly tell pvdisplay to attempt to read the md2 device for LVM info.

2
  • Thanks Dustin... doesn't look good. [root@viamao archive]# pvdisplay /dev/md2 No physical volume label read from /dev/md2 Failed to read physical volume "/dev/md2"
    – sxanness
    Jun 9, 2009 at 21:31
  • 2
    You might be right. Always keep in mind that RAID is NOT BACKUP and exists only for improved availability. This page might be helpful novell.com/coolsolutions/appnote/19386.html
    – Dustin
    Jun 9, 2009 at 21:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .