Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm on CentOS 5.2 and I'm having a problem booting my two database servers. Our IT department performed a SAN upgrade over the weekend and now I can't boot - they say the upgrade went fine but obviously something has happened. The error I get a boot time is this;

fsck.ext3: No such file or directory while trying to open /dev/VolGroup01/db

I have an external consultant who is looking at it and saying its a Superblock problem which can't be fixed, but thought these were recoverable (according to this at least; http://www.cyberciti.biz/faq/recover-bad-superblock-from-corrupted-partition/)

Anyone have any suggestions or pointers? Also, for future reference, what should I keep backups of beyond my data?


Utterly desperate and willing to pay for recovery at this point.

share|improve this question
1  
Are you sure your SAN-targets are still the same (i.e. yours)? What does "your" IT-department say about this (your are not in that department?)? I think they lost your SAN-targets or did mix them up. –  Nils Jan 22 '12 at 22:31
    
Thanks Nils, I'll ask that question. When we looked at 11pm last night they appeared to be correct. The SAN specialist we had on site suggested removing and remapping them, which we did, and which made no difference. –  sgtbeano Jan 23 '12 at 6:04
    
I'm not a part of IT no –  sgtbeano Jan 23 '12 at 7:32
add comment

3 Answers 3

I am willing to bet you know that the SAN has shifted the beginning of the physical disk off by a few bytes. I've seen this before. Its a bitch to get your files off of a disk that has done this but it is possible.

If you run 'fdisk -l' do you get messages about the starting cylinders on the device not marrying up? Its usually in brackets around each partition declaration.

Do you manage to find the LVM groups but not the disk itself? Is the LVM device made up of multiple SAN disks and just one is affected?

The following script is going to try to search for the correct offset on /dev/sdb where your lvm partition starts. No guarantees it will find anything. If it does, you might be in a good position to recover your data.

#!/usr/bin/python
import sys
def BoyerMooreHorspool(pattern, text):
    m = len(pattern)
    n = len(text)
    if m > n: return -1
    skip = []
    for k in range(256): skip.append(m)
    for k in range(m - 1): skip[ord(pattern[k])] = m - k - 1
    skip = tuple(skip)
    k = m - 1
    while k < n:
        j = m - 1; i = k
        while j >= 0 and text[i] == pattern[j]:
            j -= 1; i -= 1
        if j == -1: return i + 1
        k += skip[ord(text[k])]
    return -1

if __name__ == '__main__':
   giveup = 1024*1024*1024*2
   lba_offset = 0
   text = ""
   disk = open('/dev/sdb', 'r')
   while disk.tell() < giveup:
      #print "Checking: %f" % (lba_offset/(1024*1024))
      text = disk.read(1048576)
      s = BoyerMooreHorspool("\x00\x00\x00LVM2", text)
      if s > -1:
         print "Try offset: %d" % ((lba_offset+int(s))-533)
         sys.exit(0)
      else:
         lba_offset += 1048576
   print "Unable to find LVM position!"

Can you return what output you get?

share|improve this answer
    
Happened during a SAN upgrade yep, here's the output of your command - screenshots.cc/photos/thumb/54582-ff8z6.jpg - Any help greatly appreciated as just been informed by IT that we only have partial backups –  sgtbeano Jan 22 '12 at 20:12
    
do you have the non-thumbnail image? –  Matthew Ife Jan 22 '12 at 20:15
    
Sorry, try screenshots.cc/photos/original/54582-ff8z6.jpg –  sgtbeano Jan 22 '12 at 20:21
    
or screenshots.cc/photos/original/54582-ff8z6.jpg - i seemed to have picked the worst image host for this, sorry. –  sgtbeano Jan 22 '12 at 20:27
    
What physical disks make up /dev/VolGroup00/db? –  Matthew Ife Jan 22 '12 at 20:29
show 9 more comments
up vote 3 down vote accepted

And its all back and fixed. Turns out someone had mounted the LUNs to a Windows machine in error, then removed and put them onto the CentOS VMs without thinking it would cause a problem. Each partition was labelled as a "MICROSOFT RESERVED PARTITION" - used the cached LVM setting to bring it back.

share|improve this answer
    
Can't take credit for it I'm afraid, it was one of our consultants. I'll mark this as accepted as soon serverfault will let me. –  sgtbeano Jan 24 '12 at 12:00
add comment

Boot into single-user mode and comment out the line mounting that filesystem in /etc/fstab. That should let you boot and take a look at why that filesystem won't mount.

share|improve this answer
    
I get a prompt, could I just use umount? –  sgtbeano Jan 22 '12 at 15:15
    
If you get prompt you probably already are in single user mode. –  Hubert Kario Jan 22 '12 at 15:21
    
So I've moved on a few steps, I'm certain its a superblock issues, getting this; dumpe2fs couldn't find a valid filesystem superblock –  sgtbeano Jan 22 '12 at 16:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.