Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

It's 3 a.m., and my hosting company says a 'networking issue' occurred and, in summary, they can replace the hard drive that has developed a 'fault', but they will not assist me in recovering my data off it.

I need to access and download the contents of a single folder from a (CentOS) sda3 drive, one I missed in my backups, that is, var/www/vhosts/mydomain.com/httpdocs/images/.

What can I try next?

I have logged in and tried:

rescue:~# fdisk -l

Disk /dev/sda: 1500.3 GB, 1500311977984 bytes
64 heads, 32 sectors/track, 1430809 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1        3816     3907568   83  Linux
/dev/sda2            3817        5724     1953792   82  Linux swap / Solaris
/dev/sda3            5725     1430809  1459287040   8e  Linux LVM

rescue:~# mount /dev/sda1 /mnt

rescue:~# mount
/dev/ram0 on / type ext2 (rw,errors=remount-ro)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /mnt type ext3 (rw)

rescue:~# cat /mnt/etc/fstab
/dev/sda1       /               ext3    defaults        1 1
/dev/sda2       none            swap    sw
/dev/vg00/usr   /usr            xfs     defaults        0 2
/dev/vg00/var   /var            xfs     defaults,usrquota       0 2
/dev/vg00/home  /home           xfs     defaults,usrquota       0 2
devpts          /dev/pts        devpts  gid=5,mode=620  0 0
none            /proc           proc    defaults        0 0
none            /tmp    tmpfs   defaults        0 0

rescue:~# fsck /mnt/var
fsck 1.37 (21-Mar-2005)
e2fsck 1.37 (21-Mar-2005)
fsck.ext2: Is a directory while trying to open /mnt/var
The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

rescue:~# fsck /dev/sda3
fsck 1.37 (21-Mar-2005)
e2fsck 1.37 (21-Mar-2005)
Couldn't find ext2 superblock, trying backup blocks...
fsck.ext2: Bad magic number in super-block while trying to open /dev/sda3
The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

Update

Since following the brilliant advice given below I have managed to mount the drive, but I cannot see my website directory.. This looking pretty bad now, and extremely strange. Is there anything else I can do please? The drive size still reflects that my files should be there..

rescue:~# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               vg00
  PV Size               1.36 TB / not usable 0
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              356271
  Free PE               353199
  Allocated PE          3072
  PV UUID               YeULc0-E3XN-aF29-6Odh-JWFZ-U9qY-4KvGvl

rescue:~# vgchange -a y
  3 logical volume(s) in volume group "vg00" now active

rescue:~# fsck /dev/vg00/var
fsck 1.37 (21-Mar-2005)

rescue:~# mount /dev/vg00/var /mnt/var

Then when I look at the space:

rescue:/# cd /mnt/var/www/vhosts/
rescue:/mnt/var/www/vhosts# ll
total 0
drwxr-xr-x  10 root root 88 Jul 24 16:19 chroot
drwxr-xr-x   5 root root 49 Jul 24 16:16 default

My website folder is not listed.

Update

rescue:/mnt/var/www/vhosts# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/ram0             252M  219M   33M  88% /
tmpfs                 7.9G  4.0K  7.9G   1% /dev/shm
/dev/sda1             3.7G  319M  3.4G   9% /mnt
/dev/mapper/vg00-var  4.0G  104M  3.9G   3% /mnt/var

That looks about as bad as it can get.. Am I at a total loss?

share|improve this question
1  
How do you get from "networking issue" to "broken hard drive". It's unrelated to your immediate problem, but I'd be giving the hairy eyeball to your hosting company. –  womble Jul 25 '11 at 6:06
    
"The drive size still reflects that my files should be there.." .. yes, the drive size might be the same than before but how much is actually been used? Does df -h look like healthy or do you have a mostly empty partition waiting for you to fill it (if it was mostly full before)? –  Janne Pikkarainen Jul 25 '11 at 11:34
    
Hi, I've got: Filesystem Size Used Avail Use% Mounted on /dev/ram0 252M 219M 33M 88% / tmpfs 7.9G 4.0K 7.9G 1% /dev/shm /dev/sda1 3.7G 319M 3.4G 9% /mnt /dev/mapper/vg00-var 4.0G 104M 3.9G 3% /mnt/var Cheers –  Donna Jul 25 '11 at 11:52
    
Re womble, yeah I'm pretty concerned - I phoned them twice and each time they 'looked into it' and said they would have to send off a ticket, but the third time I phoned they claimed they had no record of any previous calls or tickets raised, and, as you said, the resulting problem sounds nothing like what it started out as :/ –  Donna Jul 25 '11 at 12:19
    
The dates on var/www/vhosts/chroot and default look suspicious to me – do you recall modifying those directories yesterday? I'm wondering if the hosting company simply recreated your disk. Is this supposed to be a physical or virtual server? Did fsck say anything out-of-the-ordinary? If you do a ls -lR /mnt/var, what do the other dates and times look like? –  fission Jul 25 '11 at 13:14
show 3 more comments

migrated from stackoverflow.com Jul 25 '11 at 2:44

This question came from our site for professional and enthusiast programmers.

3 Answers

Donna: I've been following this sad issue and hoped this would be now already happily over. Since it is not, I'm throwing a desperate attempt to you: have you tried to browse /home and /usr?

Sometimes symbolic links and/or bind mounts can effectively hide the real, actual path from you - perhaps your /var/www/vhosts was previously physically stored on /dev/vg00/usr or /dev/vg00/home and just symlinked to there.

Try also mount /dev/vg00/usr and /dev/vg00/home somewhere and see if the contents were previously actually stored on there - sometimes symbolic links and/or bind mounts can effectively hide the real path from you. The /etc/fstab does not look like promising but since you are at this point desperate, you need to try every possible method, no matter you desperate they feel...

If this did not help at all, you may try to use tools like PhotoRec against that partition and see if it finds anything. If it does not, contact your ISP once more and be extremely demanding and angry.

share|improve this answer
add comment

You won't be able to fsck /dev/sda3 directly. From your example, /dev/sda3 doesn't have a filesystem on it, it has an LVM "physical volume":

rescue:~# fdisk -l
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1        3816     3907568   83  Linux
/dev/sda2            3817        5724     1953792   82  Linux swap / Solaris
/dev/sda3            5725     1430809  1459287040   8e  Linux LVM

This physical volume most likely has (at least) one "volume group", which can in turn have multiple "logical volumes":

rescue:~# cat /mnt/etc/fstab
[...]
/dev/vg00/usr   /usr            xfs     defaults        0 2
/dev/vg00/var   /var            xfs     defaults,usrquota       0 2
/dev/vg00/home  /home           xfs     defaults,usrquota       0 2

These logical volumes are where the filesystems live that you'll need to mount.

To proceed, you'll need to get the system to recognize /dev/sda3 as a physical volume. Run pvdisplay to see if the volume is detected; if you're lucky, it is, and you can use vgchange -a y to activate the volume group into /dev/vg00. That will make /dev/vg00/var available to be fsck'ed and mounted as a standard filesystem.

share|improve this answer
    
Great minds think alike. ;-) And good call on pvdisplay. –  fission Jul 25 '11 at 5:18
    
Thank you Handyman5! Perfect. The drive mounted and fsck'ed.. but I can't see my filespace on it :/ –  Donna Jul 25 '11 at 11:32
    
I see what you mean. Perhaps the vhosts directory was symlinked out of /etc, on /dev/sda1? Try using find to look for the directory you need specifically: find / -type d -name "mydomain.com" –  Handyman5 Jul 25 '11 at 12:01
    
Thank you for your kind help. I get no results when searching for the directory or any file names. –  Donna Jul 25 '11 at 12:04
    
I know of no other place to suggest you could look; I'm sorry. –  Handyman5 Jul 25 '11 at 13:06
add comment

It looks like you were using LVM on that drive; so there appears to be a volume group vg00, which itself contains several logical volumes (eg var). I don't have a copy of CentOS here, so I can't try this myself, but you need to activate the volume group before you can do anything with it (eg the fsck you likely need to do).

Try:

mount /dev/sda1 /mnt
vgchange -a y
fsck /dev/vg00/var
mount /dev/vg00/var /mnt/var
share|improve this answer
    
Beat me to it, well done :-) –  Handyman5 Jul 25 '11 at 5:13
    
Thank you fission! The drive mounted and fsck'ed.. but I can't see my filespace on it :/ –  Donna Jul 25 '11 at 11:33
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.