I'm on CentOS 5.2 and I'm having a problem booting my two database servers. Our IT department performed a SAN upgrade over the weekend and now I can't boot - they say the upgrade went fine but obviously something has happened. The error I get a boot time is this;

fsck.ext3: No such file or directory while trying to open /dev/VolGroup01/db

I have an external consultant who is looking at it and saying its a Superblock problem which can't be fixed, but thought these were recoverable (according to this at least; http://www.cyberciti.biz/faq/recover-bad-superblock-from-corrupted-partition/)

Anyone have any suggestions or pointers? Also, for future reference, what should I keep backups of beyond my data?

Utterly desperate and willing to pay for recovery at this point.

  • 1
    Are you sure your SAN-targets are still the same (i.e. yours)? What does "your" IT-department say about this (your are not in that department?)? I think they lost your SAN-targets or did mix them up.
    – Nils
    Jan 22, 2012 at 22:31
  • Thanks Nils, I'll ask that question. When we looked at 11pm last night they appeared to be correct. The SAN specialist we had on site suggested removing and remapping them, which we did, and which made no difference.
    – sgtbeano
    Jan 23, 2012 at 6:04
  • I'm not a part of IT no
    – sgtbeano
    Jan 23, 2012 at 7:32

3 Answers 3


I am willing to bet you know that the SAN has shifted the beginning of the physical disk off by a few bytes. I've seen this before. Its a bitch to get your files off of a disk that has done this but it is possible.

If you run 'fdisk -l' do you get messages about the starting cylinders on the device not marrying up? Its usually in brackets around each partition declaration.

Do you manage to find the LVM groups but not the disk itself? Is the LVM device made up of multiple SAN disks and just one is affected?

The following script is going to try to search for the correct offset on /dev/sdb where your lvm partition starts. No guarantees it will find anything. If it does, you might be in a good position to recover your data.

import sys
def BoyerMooreHorspool(pattern, text):
    m = len(pattern)
    n = len(text)
    if m > n: return -1
    skip = []
    for k in range(256): skip.append(m)
    for k in range(m - 1): skip[ord(pattern[k])] = m - k - 1
    skip = tuple(skip)
    k = m - 1
    while k < n:
        j = m - 1; i = k
        while j >= 0 and text[i] == pattern[j]:
            j -= 1; i -= 1
        if j == -1: return i + 1
        k += skip[ord(text[k])]
    return -1

if __name__ == '__main__':
   giveup = 1024*1024*1024*2
   lba_offset = 0
   text = ""
   disk = open('/dev/sdb', 'r')
   while disk.tell() < giveup:
      #print "Checking: %f" % (lba_offset/(1024*1024))
      text = disk.read(1048576)
      s = BoyerMooreHorspool("\x00\x00\x00LVM2", text)
      if s > -1:
         print "Try offset: %d" % ((lba_offset+int(s))-533)
         lba_offset += 1048576
   print "Unable to find LVM position!"

Can you return what output you get?


And its all back and fixed. Turns out someone had mounted the LUNs to a Windows machine in error, then removed and put them onto the CentOS VMs without thinking it would cause a problem. Each partition was labelled as a "MICROSOFT RESERVED PARTITION" - used the cached LVM setting to bring it back.

  • Can't take credit for it I'm afraid, it was one of our consultants. I'll mark this as accepted as soon serverfault will let me.
    – sgtbeano
    Jan 24, 2012 at 12:00

Boot into single-user mode and comment out the line mounting that filesystem in /etc/fstab. That should let you boot and take a look at why that filesystem won't mount.

  • I get a prompt, could I just use umount?
    – sgtbeano
    Jan 22, 2012 at 15:15
  • If you get prompt you probably already are in single user mode. Jan 22, 2012 at 15:21
  • So I've moved on a few steps, I'm certain its a superblock issues, getting this; dumpe2fs couldn't find a valid filesystem superblock
    – sgtbeano
    Jan 22, 2012 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.