Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

OK, our data center did a RAID rebuild (and possibly some other stuff) on our server overnight. Something failed at some point, I checked on things this morning, and it appears as if the /var partition has been entirely wiped and brought back to a default install state.

i.e., everything we had in /var/logs is gone and everything we had in /var/db/mysql is gone, it's as if that stuff has been wiped out and replaced by a default install of FreeBSD (i.e. minimal logs, no stored .gzip logs, no more databases, etc.) BAD, REALLY BAD!

Actually, looking at it more, before if I did a "df -h" I'd see a /var partition. Now, I don't see it anymore. But I can get to /var... it just doesn't show up in "df -h". I do see an entry in /etc/fstab for /var.

I have pretty much no idea what they did. I'm waiting to hear from them. So I realize that this is a somewhat unanswerable question. I'm just looking for ideas on what I can check/what might have happened, etc. until I get more details from the data center about what they actually did.

In the meantime, can anyone propose a theory for where the data in /var might have gone... and how I can check if it still exists somewhere?

EDIT We're on the right track, thanks guys!

[kp@afekan /var/log]$ sudo tail -n 10 /var/log/dmesg.today

SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/mfid0s1a
WARNING: /tmp was not properly dismounted
/tmp: mount pending error: blocks 24 files 0
WARNING: /usr was not properly dismounted
WARNING: R/W mount of /var denied.  Filesystem is not clean - run fsck
WARNING: R/W mount of /var denied.  Filesystem is not clean - run fsck
bce0: link state changed to UP
bce1: link state changed to UP

They're working on it!

FIXED! Problem solved, /var never got mounted, needed to be fsck'd. Thanks guys!

share|improve this question
    
Now would be a good time to think about setting up some backups. –  Kamil Kisiel Dec 17 '09 at 18:22
    
Oh we always have backups, and lots of them. :-) –  Keith Palmer Dec 18 '09 at 13:32

2 Answers 2

up vote 8 down vote accepted

Your /var partition isn't mounted. Probably there was an old, default install in /var; later on they might have added another volume and mounted it over.

Do ask them to check if every volume has been properly mounted; check /etc/fstab for the exact name/slice.

Also, it is also very likely that your current var partition has been created at system startup by the various daemons which, since they could not find their own directories, decided to create /var/log, /var/tmp, etc.

Say hi to the folks in the data centre :P

share|improve this answer

Please show the output of the following three commands:

# cat /etc/fstab
# mount 
# df -kh

It's possible that they have not mounted /var from /etc/fstab for whatever reason and it's living on your root directory.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.