I'm trying to write reference material for a friend who has power outages a couple of times a year which affect their linux server. This person is not well versed in linux systems administration, so I want to include screenshots and a fairly comprehensive description of what to expect, and how to respond.

One of the more common questions that comes up (a couple times a year; they're in a lightning-prone area) is how to recover from a corrupt or missing superblock in the root filesystem.

I have documentation on how to fix the problem, but it assumes you have a nodding acquaintance with linux and console-based systems administration. What I'd like to do is setup a KVM on my server, crash the root filesystem on purpose, and take screenshots of the initial discovery along with the steps to recover.

I cannot seem to trigger that missing/corrupt superblock issue. I've been trying things like:

(/dev/sda2 is where the root LVM partition lives) dd if=/dev/zero of=/dev/sda2 bs=1024 count=1 seek=1

(/dev/mapper/vg_sys-lv_root is the root LVM partition) dd if=/dev/zero of=/dev/mapper/vg_sys-lv_root bs=4096 count=1

Neither of these work; they both seem to create mayhem, but in very different ways from the run-of-the-mill missing/corrupt superblock.

Is there any way to simulate this scenario on a KVM guest (i.e. without setting up a machine and then pulling the power cord out of it)?

  • Have you considered just making sure that you have a good backup system that is easy to restore from? Then you just have to write one set of directions. – Zoredache Aug 9 '11 at 22:14
  • @Zoredache unless I'm mistaken, repairing this particular problem takes only a few minutes, while restoring the whole system could take hours. – jdcasey Aug 9 '11 at 22:18

Make sure you're deleting all superblocks, including the backup superblocks

dumpe2fs /dev/path_to_your_volume | grep -i superblock

From there, you can kick all the superblocks down like so many tinkertoys.

  • ...but am I doing the kicking correctly? I'm getting the list of superblock locations using the command you mentioned, and getting the block size using: dumpe2fs /dev/foo | grep -i 'block size' ...if I'm doing all that correctly, is the dd command I'm using correct to knock out the superblock and trigger that error message? In my testing, if I knock out the 0-offset on the LVM root partition (0-offset being the first superblock it lists) it goes pretty haywire and panics...no chance to fix it. – jdcasey Aug 9 '11 at 22:46
  • 1
    @John Are you blowing it up with the filesystem mounted? If you're needing to reproduce a power outage that screws up the superblock, you may need to do your artificial corruption while booted to other media. – Shane Madden Aug 9 '11 at 23:00
  • @jdcasey If you're overwriting all superblocks on a volume, then you're certainly hosing the filesystem right proper. Is it exactly what needs to be done to get the error you want? In theory, it should certainly give you a missing superblock error. – Wesley Aug 10 '11 at 1:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.