How do I recover from a failed fsck?

Question

I ran fsck and now my Linux server (Oracle Enterprise Linux (a Red Hat derivitave)) will not boot. The old blocks are in the lost+found folder.

How do I get the box working again without rebuilding?

-- Edit -- Well... After spending 4 days having the system looked at, we decided to rebuild. It may actually be good that this happened. We've identified several pieces of documentation that were missing. We found a version incompatibility with our storage device.

Best of all, we were able to get most of the long-term maintenance of this server moved back to the operations group. (We made them do the actual reinstall, while we were there to answer questions)

Answer 1

You have a mess, basically. If the filesystem will mount read-only you should be looking at copying all the data out of the partition that you want to salvage and starting fresh. There is no "magic" command you can execute to clean up an inconsistent filesystem. fsck tried to help, but if things are so bad that fsck can't put them back together again, short of you examining the filesystem metadata by hand and attempting to put it back together again yourself, I think your best hope is to get what data you can out and cut your losses.

I'd recommend attaching a second hard disk drive w/ enough space to hold an image of the original disk and imaging the original disk off with dd. Then you can reinstall, mount the image you took of the original disk read-only, and start copying data off of it.

Backups are your friend. Look at this as a valuable lesson.

Answer 2

Some links that may help:

Saving data from broken opensuse partition

Here's a pdf that could be good.

and, a ubuntu forum post that has a lot of links inside that might point you in the right direction.

HTH..