Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I would like to rescue data on my software raid-6 array. I did some stupid actions (described bellow) with this original array.

Main Question:

I need to know if original data stored on raid-6 array are definitely lost (or not) after the following actions have been prepared upon this array (executed in the order listed bellow):

  1. zeroing super-blocks of all the active disks/partitions registered in the array

  2. executing the "mdadm --create ..." command using different options (see bellow for list) than have been used when array have been created originally: -> different chunk size -> different layout -> different disks order

  3. resync-ing the array

Note: Specific values of mdadm parameters should not be relevant here, since this is about the principle how mdadm works ...

I think points 1) & 2) should not even touch the original data since they are supposed to manipulate just superblocks

I see the 3) point as most critical from data lost point of view: I'm not sure what exactly is happening with array during resync, but based on heavy activity of all involved hard drives (for ~7 hours) I assume the data storage area is completely re-processed ...

Sub-questions:

  1. Does ordering of hard drives/partitions (as they are ordered on mdadm command line) play role for raid6 creation & initial resync?

  2. What everything is necessary to backup after array creation to be able to safely re-create the array in similar situation as my is (e.g. backup superblocks informations & partition table info for each disk involved in the array ...)?

Comment:

The mdadm wiki article (http://en.wikipedia.org/wiki/Mdadm) should be revised and author should be kicked in to ass a little bit, or more then just a little bit ...
The article mentions the zeroing superblocks & subsequent array re-creation as solution for getting rid of the "mdadm: Cannot open ...: Device or resource busy" issue.
Author somehow forgot to mention important step - to backup the parameters the original array (superblocks) as the first step ... and also my investigation seems to point out that the ordering of the involved disks/partitions plays role as well ...

Thanks for answers,
Peter

share|improve this question
add comment

3 Answers

You're probably right about only affecting the superblocks until the drives began to re-sync, but at this point that's hardly a significant detail. By now your data is in an unknown state if it's even there at all.

I don't think you'll be helped any further by advice you read on the Internet, even from this site. Simply trying out things you read online is only a valid course of action if failure is not a problem.

If the data is very important to you, I'd recommend turning the drives over to a data recovery lab and letting them work on it. If your data is still intact in some form, they'll have the experience to know how to fix your problem without causing any further damage.

Edit
Luckily, since rebuilding your parity data should ONLY affect the parity data, you theoretically may still have just enough information left untouched to reconstruct your original content, assuming you started out with a clean and healthy array before this mess happened.

However, reconstructing it would be non-trivial and require a lot of work building a profile of the array by hand, especially if you changed some of the dimensions of your array when you overwrote the superblocks.

It would almost definitely not survive doing this sort of thing twice, though.

share|improve this answer
    
Thank you for answer –  schanhorst Oct 19 '10 at 3:00
    
I have been re-creating the array with "--assume-clean" option (e.g. "mdadm --create --assume-clean ...") in effort to find out the right superblocks configuration to avoid automatic trigger of resync -> this should ensure that just superblocks will be affect and data storage area will remain untouched ... But the last time I have been playing with --create command I forgot to specify the "--assume-clean" option and I it was to late when I recognized, that "resync" has been triggered automatically and running long enough to destroy data ... –  schanhorst Oct 19 '10 at 3:08
    
Edit w/ more info –  tylerl Oct 19 '10 at 5:59
add comment

The confusing part is:

create and then resynced for 7 hours.

When an array is created, the filesystem is generally created on top and there is no resyncing. Zeroing the superblocks makes it difficult for the raid set to figure out which order the drives were in when the raidset tries to assemble itself automatically. What did you do with the create? create it with 2 drives, missing 2, then grow it and add the others? or did you create it as a raid-6 set and add all drives at the same time.

If the latter, it should not have resynced. IF you didn't grow the array and the drives resynced, it sounds almost like it detected a portion of the array. After the resync, did you have to make a filesystem or did a filesystem already exist?

If the chunk size changed, that's not good. If the alignment of the partitions changed, again, not good. Order of the drives changed, again, not good. You have a number of things that probably have made it extremely difficult to recover the data if at all.

The one thing that I find very curious is the resync. Raid-6 can take two drive failures and still have the ability to recover. But, some of the things you did almost make me think you're in a situation that is going to be almost unrecoverable except by experts.

share|improve this answer
    
The array is raid-6 assembled from 6 1TB disks, what gives ~4TB array data space. I created have been re-creating the array using the same number of disks (partitions) as the original array was created, executing the "mdadm --create --assume-clean ... /dev/md0 /dev/sd[abcdefg]1". I varied chunk size, layout and disk order from execution to execution. I tried to mount the /dev/md0 after each execution of "mdadm --create --assume-clean ..." & "mdadm --assemble ...". I have been trying until it once happened that I forgot to specify "--assume-clean" option, what autom. triggered resync. –  schanhorst Oct 19 '10 at 3:54
    
Basically I did following steps –  schanhorst Oct 19 '10 at 4:21
    
(the original array has been working without any issues ... ) "mdadm --misc --zero-superblock /dev/sd[abcdef]1" (then x-times following set of commands with different chunk, layout and disks order) "mdadm --create --assume-clean ... /dev/md0 /dev/sd[abcdef]1" "mdadm --assemble /dev/md0 /dev/sd[abcdef]1" "mount /dev/md0 /mnt/array" I executed the "mount ..." command to check if the array has been re-created correctly. –  schanhorst Oct 19 '10 at 4:26
    
The issue about disk order is, that the /dev/sdxy block device filenames are paired to real devices automatically during each system startup by udev functionality of the linux kernel, what is by default enabled in Ubuntu distribution ... The result is, that once you zero array superblocks and then reboot, you don't have a glue how the real hard drives were mapped before to /dev/sdxy block device files ... (order changes each restart thanks to udev ... –  schanhorst Oct 19 '10 at 4:37
    
Thanks for answer ... I also think the data are gone ... But I'm still curious what is happening during the resync procedure ... I also remember, that the the first time I created the original array, the resync procedure took also several hours ... –  schanhorst Oct 19 '10 at 4:46
show 9 more comments

I looked at the wikipedia article, and the bit about deleting the superblock etc. is right, but it's supposed to be used for solving a different problem than what you were apparently trying to solve. The intention in the article is to "clean out" a drive that has old RAID settings on it so that you can use it in a new array.

You should, of course, never clear the superblock on a drive that currently contains data that you want to preserve. The superblock contains critical information regarding how the data on that drive is organized, so erasing or changing the superblock it is unhelpful unless you intend to throw away the data with it.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.