OK, our data center did a RAID rebuild (and possibly some other stuff) on our server overnight. Something failed at some point, I checked on things this morning, and it appears as if the /var partition has been entirely wiped and brought back to a default install state.

i.e., everything we had in /var/logs is gone and everything we had in /var/db/mysql is gone, it's as if that stuff has been wiped out and replaced by a default install of FreeBSD (i.e. minimal logs, no stored .gzip logs, no more databases, etc.) BAD, REALLY BAD!

Actually, looking at it more, before if I did a "df -h" I'd see a /var partition. Now, I don't see it anymore. But I can get to /var... it just doesn't show up in "df -h". I do see an entry in /etc/fstab for /var.

I have pretty much no idea what they did. I'm waiting to hear from them. So I realize that this is a somewhat unanswerable question. I'm just looking for ideas on what I can check/what might have happened, etc. until I get more details from the data center about what they actually did.

In the meantime, can anyone propose a theory for where the data in /var might have gone... and how I can check if it still exists somewhere?

EDIT We're on the right track, thanks guys!

[kp@afekan /var/log]$ sudo tail -n 10 /var/log/dmesg.today

SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/mfid0s1a
WARNING: /tmp was not properly dismounted
/tmp: mount pending error: blocks 24 files 0
WARNING: /usr was not properly dismounted
WARNING: R/W mount of /var denied.  Filesystem is not clean - run fsck
WARNING: R/W mount of /var denied.  Filesystem is not clean - run fsck
bce0: link state changed to UP
bce1: link state changed to UP

They're working on it!

FIXED! Problem solved, /var never got mounted, needed to be fsck'd. Thanks guys!

  • Now would be a good time to think about setting up some backups. – Kamil Kisiel Dec 17 '09 at 18:22
  • Oh we always have backups, and lots of them. :-) – Keith Palmer Jr. Dec 18 '09 at 13:32

Your /var partition isn't mounted. Probably there was an old, default install in /var; later on they might have added another volume and mounted it over.

Do ask them to check if every volume has been properly mounted; check /etc/fstab for the exact name/slice.

Also, it is also very likely that your current var partition has been created at system startup by the various daemons which, since they could not find their own directories, decided to create /var/log, /var/tmp, etc.

Say hi to the folks in the data centre :P


Please show the output of the following three commands:

# cat /etc/fstab
# mount 
# df -kh

It's possible that they have not mounted /var from /etc/fstab for whatever reason and it's living on your root directory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.