Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I received an email yesterday that one of our users was trying to make room for a heartbeat/clustering package which requires its own partition to act as a voting disk. To do this, he attempted to reduce the size of the root partition's logical volume, and then create a new logical volume for this purpose. However, he forgot to resize the filesystem first (or include the -r switch in the command). He also forgot to unmount the root partition by running this process from a rescue cd. The system is now refusing to boot into the OS with the following error:

Either the superblock or the partition table is likely to be corrupt! Unexpected Inconsistency; run fsck manually.

The system them drops the user into single user mode.

Is it possible to rescue the filesystem, or is it hosed? Its running ext3.

share|improve this question
add comment

3 Answers

up vote 3 down vote accepted

If a filesystem has already been initialised on the new LV then you're probably hosed, because data will have been overwritten. You can still try to recover it, but I wouldn't be too hopeful. If a filesystem hasn't been initialised then it's theoretically possible to recover it (but I haven't tried this myself).

Firstly, make a backup of the entire drive, you'll need it in case the recovery goes wrong. The next step is to try find the LVM metadata backups which LVM creates in /etc/lvm/archive before making changes. If the root volume isn't accessible you can try running e2fsck to get it mountable, and hope that the backup files were stored near the beginning of the volume. If you do this you'll want to continue recovery from before you ran e2fsck (i.e. restore from the backup after getting the backup files).

If you managed to get a backup file, restore the LVM configuration with vgcfgrestore. If you couldn't get a backup file you're just going to have to hope that the initial volume was completely sequential. Remove the new LV and then extend the old LV to its original size.

Once you've got the old LV back to the correct size, cross your fingers and run e2fsck. And you really need to make that backup first, you're probably not going to get this right on the first attempt.

share|improve this answer
    
Thank you for your comment, it is very much appreciated. At this point I think it is better to rebuild. Luckily this is still a fairly new server so we're not going to lose more than a software install. We can have it up and running in a few hours from scratch. It's actually easier than trying to get rescue media at this point. Thanks again! –  Matthew Jun 19 '12 at 13:26
    
Holy cow I was able to resurrect the box! I tried running through the last few steps (remove the new lv, extend the old lv. After a reboot fsck kicked in, said it found errors, scanned everything and made repairs. Checking now but it looks like the os is up and functional –  Matthew Jun 20 '12 at 20:25
add comment

What I would try:

  • Use a rescue system, eg. GRML or an Ubuntu live system.
  • Resize the volume back to it's original size and hope that LVM assigns the same physical disk space. This should work if no other modifications have been done to the volume group in the mean time.
  • Make an image, or at the very least, a snapshot of the volume.
  • Try to fix it with fsck.
  • Hope for the best

In all likelihood, I would assume the file system to be dead but this might help with a awful lot of luck.

share|improve this answer
    
Thank you for your comment, it is very much appreciated. At this point I think it is better to rebuild. Luckily this is still a fairly new server so we're not going to lose more than a software install. We can have it up and running in a few hours from scratch. It's actually easier than trying to get rescue media at this point. Thanks again! –  Matthew Jun 19 '12 at 13:26
add comment

Today I was that user... Fortunately it was on a testing environment, so I kept a clear thought process ;)

I was able to resolve the error by reversing what I had done. I reduced my home lvm by 100G, did not resize, then expanded my root lvm by 100G and still did not resize the FS.

I booted up from a liveCD (CentOS 6.4), shrank the boot by 100G, resized, expanded the home by 100G, resized. Rebooted, and everything was back to normal.

share|improve this answer
    
StackOverflow is a question/answer format. Please read the FAQ: stackoverflow.com/faq –  tristan Jul 10 '13 at 21:30
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.