To make a long story short, I am responsible for a media server that has two RAID arrays (/dev/md0 and /dev/md2). I still have a ton to learn about software RAID and LVM, etc. We had a drive fail in /dev/md0, okay not a big deal I replaced the bad drive, reassembled the RAID array (which took most of last night to complete) and this morning I came in and was able to mount /dev/md0 with no data loss.
Little did I know, something became corrupt with /dev/md2. Now /dev/md0 was set-up by my co-worker two years ago, I set-up /dev/md2 about six months ago. Always having done RAID using the hardware controllers built into servers from Dell and HP, this is the first time I have had to deal with software RAID. I followed the instructions on [url]http://www.gagme.com/greg/linux/raid-lvm.php[/url] which I felt was the best idea as LVM would give me the ability to grow the RAID array in the future (I just now learned that mdadm can do the same). There was no issues with this until today. After rebooting the machine after it repaired /dev/md0 it was unable to mount /dev/lmv-raid/lvm0 (said the file or directory couldn't be found). It almost looks like the logical volume is gone. I do not understand what would have caused this. No drives in the second array were bad, I didn't touch the second array. Nothing should have gone bad with the array. But it looks like something isn't right.
Now /dev/md2 is running and is in a clean state, but I cannot mount it. I am afraid that data is gone. The really terrible thing is that there is no explanation in my mind as to what happened. I am assuming that I can't mount /dev/md2 because the RAID array is expecting to be controlled by LVM. Maybe I do not understand the system well enough, but if I have a clean array shouldn't the data still be in the array? Any idea how I can go about recovering the data? The last thing I want to do is have to go back to my boss and tell him that in the process of fixing one array another array magically broke and all your data is gone. I have looked at multiple sites, but since I don't know what actually broke with it all I don't know if I have really lost data or if there is something else that I am missing.
Any advice would be greatly appreciated, I have been struggling with this all morning long and am at a total loss.
Here is some more information:
[root@viamao ~]# mdadm --detail /dev/md2
/dev/md2:
Version : 00.90.01
Creation Time : Tue Jun 9 11:32:06 2009
Raid Level : raid5
Array Size : 4395415488 (4191.79 GiB 4500.91 GB)
Device Size : 1465138496 (1397.26 GiB 1500.30 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 2
Persistence : Superblock is persistent
Update Time : Tue Jun 9 15:54:39 2009
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
0 8 128 0 active sync /dev/sdi
1 8 144 1 active sync /dev/sdj
2 8 160 2 active sync /dev/sdk
3 8 176 3 active sync /dev/sdl
UUID : 88bbe155:b6d852b1:3ad8a77d:5eba3ee2
Events : 0.36
[root@viamao archive2]# mount -t ext3 /dev/md2 /share/archive2
mount: wrong fs type, bad option, bad superblock on /dev/md2,
or too many mounted file systems
[root@viamao archive2]# mount
/dev/hda3 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/md0 on /share/archive1 type ext3 (rw)
[root@viamao share]# cat /proc/mdstat
Personalities : [raid5]
md2 : active raid5 sdl[3] sdk[2] sdj[1] sdi[0]
4395415488 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
md0 : active raid5 sda[0] sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1]
5128021248 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]
unused devices: <none>
Anything else I can show you that might be able to help? I have a active RAID array that I can't mount, thinking this has something to do with the fact that this used to be on lvm, but now when I try to mount /dev/lvm-raid/lvm0 it tells me the special device cannot be found. lvdisplay doesn't give me anything pvdisplay only shows me the drive that my OS is on. vgdisplay also yields nothing.
Thoughts? I am seriously at a loss here.