0

I have a Debian squeeze installation with 6x 3TB Disks.

On the SW RAID5 are 4 partitions.

  • First for gpart BIOS,
  • RAID 1 for "/",
  • RAID5 for swap
  • and the biggest part RAID5 for files.

While the system was being build all disks lay on the open case. During this I saw that one disk could possibly fall of the case and I moved two a bit. While doing this both disk spins down, sorry the system runs, and the RAID crashes with the error: error can not start while 2 missing disks. 

mdadm: /dev/md2 assembled from 4 drives and 1 spare 
- not enough to start the array."

At this point I had no writeoperations. I saved some information and figure out that disk sdd4 and sdf4 failed and in my opinion sdd fails before sdf:

mdadm -E /dev/sda4
/dev/sda4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e84f0346:3f5ff3f1:507b6f9c:0fa02c63
           Name : mfsnode1:2  (local to host mfsnode1)
  Creation Time : Tue Feb  5 17:44:45 2013
     Raid Level : raid5
   Raid Devices : 6

 Avail Dev Size : 5842757597 (2786.04 GiB 2991.49 GB)
     Array Size : 29213772800 (13930.21 GiB 14957.45 GB)
  Used Dev Size : 5842754560 (2786.04 GiB 2991.49 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 4f8851b4:001bf0c0:3aab60e0:b2c5558f

    Update Time : Tue Feb  5 17:44:45 2013
       Checksum : c0376a50 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAA. ('A' == active, '.' == missing)

mdadm -E /dev/sdb4

/dev/sdb4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e84f0346:3f5ff3f1:507b6f9c:0fa02c63
           Name : mfsnode1:2  (local to host mfsnode1)
  Creation Time : Tue Feb  5 17:44:45 2013
     Raid Level : raid5
   Raid Devices : 6

 Avail Dev Size : 5842757597 (2786.04 GiB 2991.49 GB)
     Array Size : 29213772800 (13930.21 GiB 14957.45 GB)
  Used Dev Size : 5842754560 (2786.04 GiB 2991.49 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : c2f63fa7:768e9945:64826929:6f1f68c2

    Update Time : Tue Feb  5 17:44:45 2013
       Checksum : b3ea7d20 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAA. ('A' == active, '.' == missing)

mdadm -E /dev/sdc4

/dev/sdc4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e84f0346:3f5ff3f1:507b6f9c:0fa02c63
           Name : mfsnode1:2  (local to host mfsnode1)
  Creation Time : Tue Feb  5 17:44:45 2013
     Raid Level : raid5
   Raid Devices : 6

 Avail Dev Size : 5842757597 (2786.04 GiB 2991.49 GB)
     Array Size : 29213772800 (13930.21 GiB 14957.45 GB)
  Used Dev Size : 5842754560 (2786.04 GiB 2991.49 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : e9861f3e:4de4d0ce:7d4b6dd7:e1215fc7

    Update Time : Tue Feb  5 17:44:45 2013
       Checksum : 86fc2eab - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAAA. ('A' == active, '.' == missing)

mdadm -E /dev/sdd4

/dev/sdd4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 7b99380e:51d754cf:921c68e9:7b830d6a
           Name : mfsnode1:2  (local to host mfsnode1)
  Creation Time : Tue Feb  5 17:06:37 2013
     Raid Level : raid5
   Raid Devices : 6

 Avail Dev Size : 5842757597 (2786.04 GiB 2991.49 GB)
     Array Size : 29213772800 (13930.21 GiB 14957.45 GB)
  Used Dev Size : 5842754560 (2786.04 GiB 2991.49 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 0da58625:14ed8675:6a7c4ba4:337d8c4b

    Update Time : Tue Feb  5 17:06:37 2013
       Checksum : 5f97164a - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAA.AA ('A' == active, '.' == missing)

mdadm -E /dev/sde4

/dev/sde4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e84f0346:3f5ff3f1:507b6f9c:0fa02c63
           Name : mfsnode1:2  (local to host mfsnode1)
  Creation Time : Tue Feb  5 17:44:45 2013
     Raid Level : raid5
   Raid Devices : 6

 Avail Dev Size : 5842755584 (2786.04 GiB 2991.49 GB)
     Array Size : 29213772800 (13930.21 GiB 14957.45 GB)
  Used Dev Size : 5842754560 (2786.04 GiB 2991.49 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : b70cd4f6:1594cc29:b4346929:89a5ed34

    Update Time : Tue Feb  5 17:44:45 2013
       Checksum : 5a36c944 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAA. ('A' == active, '.' == missing)

mdadm -E /dev/sdf4

/dev/sdf4:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : e84f0346:3f5ff3f1:507b6f9c:0fa02c63
           Name : mfsnode1:2  (local to host mfsnode1)
  Creation Time : Tue Feb  5 17:44:45 2013
     Raid Level : raid5
   Raid Devices : 6

 Avail Dev Size : 5842755584 (2786.04 GiB 2991.49 GB)
     Array Size : 29213772800 (13930.21 GiB 14957.45 GB)
  Used Dev Size : 5842754560 (2786.04 GiB 2991.49 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 06202661:79792af2:6c8d02ae:769bdded

    Update Time : Tue Feb  5 17:44:45 2013
       Checksum : ca70109c - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAA. ('A' == active, '.' == missing)

I think the superblock is ok, the chunksize the same and all other parameter looks good too. After that I started to test some variations:

Test1:  
mdadm --create --level 5 -n 6 --chunk=512 --assume-clean /dev/md2 \
                /dev/sd{a,b,c,d,e,f}4
-> filesize 708MB with 20603326 lines and canceling at the end by e2fsck
- bad superblock or partition table is damage
- bad checksum of group or descriptor
- lots of invalid inodes
- canceled with lots of illegal blocks in inodes

Test2:

mdadm --create --level 5 -n 6 --chunk=512 --assume-clean /dev/md2 /dev/sd{a,b,c,d,e}4 \
                missing
-> filesize 1,3GB  with 37614367 lines and canceling by e2fsck at the end
- back to original superblock
- bad superblock or damaged partition table at the beginning
- lots of invalid inodes
- canceled with iteration of inode

Test3:

mdadm --create --level 5 -n 6 --chunk=512 --assume-clean /dev/md2 /dev/sd{a,b,c}4 \
                missing /dev/sd{e,f}4
-> filesize 1,4GB with 40745425 lines and canceling by e2fsck at the end
- errors see test2
- read error while reading next inode

Test4:

mdadm --create --level 5 -n 6 --chunk=512 --assume-clean /dev/md2 \
                /dev/sd{a,b,c,f,e,d}4
->filesize 874MB with 25412000 lines and break by e2fsck at the end
- try original superblock
- bad superblock or damaged partitiontable
- than lots of checksumm  invalid deskriptor of group
- at the end illegal block in inode to much invalid blocks in inode

Test5:

mdadm --create --level 5 -n 6 --chunk=512 --assume-clean /dev/md2 /dev/sd{a,b,c}4 \
                missing /dev/sd{e,d}4
-> filesize 1,6GB with 45673505 lines and canceling at the end by e2fsck

Test6:

mdadm --create --level 5 -n 6 --chunk=512 --assume-clean /dev/md2 /dev/sd{a,b,c,f,e}4 \
                missing
- try original superblock
- bad superblock or damage partition table
- lots of checksum error in group descriptor
- ends with conflict in inode table with another filesystem block
-> filesize 542MB with 15727702 lines and canceling at the end by e2fsck

Teset6 looks like the best one, but what do you think and perhaps what could I could do otherwise?

I think the best could be the last but I am absolutely unsure.

Please help for further investigation and hopefully recovery of my data.

Improve this question
7
  • 1
    What do you mean with "While the system was being build all disks". DO you mean you was just installing it? In that case, just wipe and reinstall and restore the data from backup. That is probably much faster.
    – Hennes
    Mar 2, 2013 at 20:30
  • 7
    Using six 3TB drives in RAID-5 is begging to lose data. serverfault.com/questions/192995/…
    – Evan Anderson
    Mar 2, 2013 at 20:33
  • Test 6 looks closest to the 'Active Device' from your Mdadm -E outputs (counting from 0 to 5 via partitions a4/b4/c4/f4/e4/d4.) Still, if you have backups: use them. And before you recreate think about what you are doing and why. RAID 5 can be useful in some situations. It is not the right thing in all of them.
    – Hennes
    Mar 2, 2013 at 20:50
  • The system is private and i build on it if i have time. thats why the case is still open. that was mentioned with "while the system.." and the raid and all was good and runs. there was no rebuild or write action on it, the power was just on.
    – Sunghost
    Mar 2, 2013 at 21:09
  • i know this facts, but as a think about it, it was a solution with max of diskcapacity and probably a bit failure tollerance. i know all the facts, but as long as i didnt win some money, i cant by a backupsolution for 40+TB i also had build up a raid10. ok forget all these things please and help me to probably rescue the raid..ok you think i should give :"mdadm --create --level 5 -n 6 --chunk=512 --assume-clean /dev/md2 /dev/sd{a,b,c,f,e}4 missing" a chance? after that i would mount the raid and check the files?perhaps i think about a raid6 solution but in mind to loose 3 and up more diskspace
    – Sunghost
    Mar 2, 2013 at 21:15

1 Answer 1

Reset to default
0

If you have lost two drives in your RAID 5 array, there is no recovery from that at the RAID array level, the array has been lost. You will need to resort to attempting to get the drives working again, which isn't very likely. There are drive recovery tools, like R-Studio, with which you might have a little luck. But most likely you will need to send your disks to a forensic shop that can do platter recovery. This is very costly, however.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.