Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a software raid 10 with 6 x 2tb hard drives (raid 1 for /boot), ubuntu 10.04 is the os.

I had a raid controller failure that put 2 drives out of sync, crashed the system and initially the os didnt boot up and went into initramfs instead, saying that drives were busy but I eventually managed to bring the raid up by stopping and assembling the drives.

The os booted up and said that there were filesystem errors, I chose to ignore because it would remount the fs in read-only mode if there was a problem.

Everything seemed to be working fine and the 2 drives started to rebuild, I was sure that it was a sata controller failure because I had dma errors in my log files.

The os crashed soon after that with ext errors.

Now its not bringing up the raid, it says that there is no superblock on /dev/sda2, even if I assemble manually with all the device names.

I also did a memtest and changed the motherboard in addition to everything else.

EDIT: This is my partition layout

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x0009c34a

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *        2048      511999      254976   83  Linux
/dev/sdb2          512000  3904980991  1952234496   83  Linux
/dev/sdb3      3904980992  3907028991     1024000   82  Linux swap / Solaris

All 6 disks have the same layout, partition #1 is for raid 1 /boot, partition #2 is for raid 10 far plan, partition #3 is swap, but sda did not have swap enabled

EDIT2: This is the output of mdadm --detail /dev/md1

Layout : near=1, far=2
Chunk Size : 64k

UUID : a0feff55:2018f8ff:e368bf24:bd0fce41
Events : 0.3112126

Number Major Minor RaidDevice State
0      8     34    0          spare rebuilding /dev/sdc2
1      0     0     1          removed
2      8     18    2          active sync /dev/sdb2
3      8     50    3          active sync /dev/sdd2
4      0     0     4          removed
5      8     82    5          active sync /dev/sdf2

6      8     66    -          spare /dev/sde2

EDIT3: I ran ddrescue and it has copied everything from sda except a single 4096 byte sector that I suspect is the raid superblock

EDIT4: Here is some more info too long to fit here

lshw: http://pastebin.com/2eKrh7nF

mdadm --detail /dev/sd[abcdef]1 (raid1): http://pastebin.com/cgMQWerS

mdadm --detail /dev/sd[abcdef]2 (raid10): http://pastebin.com/V5dtcGPF

dumpe2fs of /dev/sda2 (from the ddrescue cloned drive): http://pastebin.com/sp0GYcJG

I tried to recreate md1 based on this info with the command

mdadm --create /dev/md1 -v --assume-clean --level=10 --raid-devices=6 --chunk=64K --layout=f2 /dev/sda2 missing /dev/sdc2 /dev/sdd2 missing /dev/sdf2

But I can't mount it, I also tried to recreate it based on my initial mdadm --detail /dev/md1 but it still doesn't mount

It also warns me that /dev/sda2 is an ext2fs file system but I guess its because of ddrescue

EDIT5: I figured out /dev/sdb was actually /dev/sdc and vice versa and I also made a major mistake since the newer live cd I was using defaulted to 1.2 metadata which has probably destroyed the data on these disks, good thing is I cloned them to 6 other drives.

EDIT6: I finally managed to recover my array by recreating it again, this time with the correct --metadata option. If someone is in my situation and reading this, its very important to mirror your data so you can recover from a mistake like I did, also make sure to mdadm --examine all drives because their names will get changed when booting a live cd and its important to recreate in the correct order, any drives marked as spares have to be set as missing.

share|improve this question
    
Can you give your full partition/disk layout? –  Andrew Dec 4 '12 at 2:47
    
I agree with the previous poster. We need to see/understand how many disks you have, how the disks are configured into arrays and what the partition table for the arrays look like. You are showing sdb in you partition scheme...but, the rest of your post describes sda. You also mention RAID 1 for /boot. Unless you have some overwhelmingly huge need for a gigantic partition, I would recommend against having stripe sets on large drives. I take it you know about not using RAID 5 or RAID 6 (because of size and error rates making recoveries potentially impossible) for large partitions. Were the 2 dr –  jim_m_somewhere Dec 4 '12 at 16:44
    
It seems the 2 drives were different mirrors because they were both rebuilding, and the initial initramfs was because all drives were reporting that they were busy. I will try to post a full mdadm detail. –  Shoshomiga Dec 4 '12 at 18:31

1 Answer 1

up vote 1 down vote accepted

As the owner of professional data recovery firm the proper way to rebuild a broken RAID like this would be to create clones or images of each drive. Never work with the originals. Once you have images of the drives you would need to recreate the RAID in some kind of software like X-Ways. Be sure to keep the correct order of the drives! It can be quite complicated rebuilding RAID ARRAYS, and if the data is important you should label the drives and send them out to a reputable company, but most RAID recoveries are not cheap.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.