Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I've just booted to find my software RAID 5 in Ubuntu not mounting. When trying to mount it gave me an NFS error (which was confusing). I ran fsck on /dev/md0 and my screen scrolled with fixes for about an hour. It claimed to be complete, however I've mounted it and the folder structure is empty. It just has a lost+found folder containing hundreds of files like the screenshot below:

Very confusing files!

share|improve this question
1  
This is why RAID is not backup. –  David Schwartz May 6 '12 at 23:11
    
I didn't say it was backup, I'm asking what could have caused this and ideas to repair it. Failing that ways to check it won't happen again. –  SimonJGreen May 7 '12 at 7:47
    
RAID doesn't protect against filesystem corruption, it only replicates it very quickly across drives. If your memory was failing, if an application or bug reared its head, if a drive started going nutters and it replicated the error across other disks, etc. then you're going to have issues. RAID only helps protect against a straightforward physical drive dying. It won't save you from filesystem-level corruption. –  Bart Silverstrim May 7 '12 at 12:15
    
Unless you're well versed in trying to recover data from the block level and navigating inodes, or unless you get very lucky running the "file" command and figuring out what those fragments of files are so you can piece them back together, your filesystem is purdy hosed. You can either hire a company to recover the data or write it off if there's no backup. You might find a filesystem guru who can piece things back together but you're going to pay quite a bit for it. –  Bart Silverstrim May 7 '12 at 12:19
    
You may want to run the same machine through memtest86. Nothing corrupts a filesystem or data faster than a bad spot of RAM that is being used as a buffer. Having detected and sent back 2 sticks of RAM (and I'm on a 3rd now) I can say that consumer-grade memory isn't what it's always cracked up to be. Hopefully you have ECC RAM. –  Avery Payne Jan 26 at 3:15

1 Answer 1

up vote 5 down vote accepted

Looks like the filesystem was hosed and the fsck didn't fully repair it.

At this point I'd be tempted to check the logs to see if the disks are all physically working (noises? SMART status? errors in the logs regarding resets? etc.) and restore from backup rather than spend more time trying to straighten out the results of the fsck.

share|improve this answer
    
Cheers for the idea, however this was a 4TB NAS used for backup, mdadm shows all disks in the array are fine, all the disks SMART ok. 90% of the data can be recovered from else where but there is a good 150-200GB that was moved from working machines to here as central storage. –  JTotham May 6 '12 at 22:08
2  
Little confused...if the NAS was used for backup, that implied there is another copy of the data somewhere to rebuild from? But if this is a central file sharing server is there any tape backup or drive backup that the server is being backed up to? –  Bart Silverstrim May 6 '12 at 22:56
    
This was a home NAS used mostly for backup but also as shared storage so a tape library isn't really practical. There is no 2nd backup of the main data and no backup at all of the shared data, my problem i know and one that i was in the process of fixing. Once rebuilt / recovered i am going to setup some online backup space that the shared data can be replicated too. The most annoying thing is that its was fine 2 days ago and then the filesystem just died with no warning. –  JTotham May 7 '12 at 6:48
    
Smart reports ok. Can you explain what you mean by "noises"? –  SimonJGreen May 7 '12 at 7:46
    
Hard drives physically failing but still nearly working still often have the controller/OS send resets to the drive to correct for read/write errors. You can hear a series of clicks or thunks from the drive motor when they do this. –  Bart Silverstrim May 7 '12 at 12:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.