Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

on our production server, there is a small drive for the main mount point "/" /var/log is taking much space and I have to manually delete some files. How can I move /var/log/ to let's say /home/log WITHOUT REBOOTING.

Here is the thing I thought :

$mkdir /home/log

$rsync -a /var/log /home/log

$mount --bind /home/log /var/log

$/etc/init.d/rsyslof/restart

BUT : I know that some services use file descriptors, so they'll continue to use /var/log or inodes. Thanks

share|improve this question

3 Answers 3

up vote 17 down vote accepted

Proper design

I assume you are unable to simply extend the filesystem in question (using lvextend && ext2online), because you do not use LVM or use wrong filesystem type.

Your approach

What you've proposed might work if you signal the daemons with SIGHUP (kill -1 pid). Obviously you would need to later on "mount -o bind / /somewhere" and clean up what has been left underneath mounted /var/log. But it has a bad smell for me, especially for production.

Avoid downtime, have a clean result (but complicated to do)

Forget about "mount -o bind" idea, create a new LV/partition, but don't mount it yet.

lsof | grep /var/log             # lists open files in /var/log

For each daemon that has any open file (I would expect at least syslog, inetd, sshd):

  • reconfigure the daemon no to log to /var/log
  • refresh the daemon (kill -1 or /etc/init.d/script reload)
  • confirm with lsof | grep /var/log that daemon has closed its files

Mount over /var/log. Restore old configurations, SIGHUP/reload daemons again.

Easy way (downtime)

Create a new LV/partition and mount it properly over either /var or /var/log. The easy way is to take down the server to maintenance mode (single-user mode), and use the actual console (not ssh) for the operation.

share|improve this answer

Another thing that you could do is:

  • Stop the processes that have open files on /var/log
  • Verify that there aren't any processes with open files on /var/log (using lsof as kubanskamac suggested)
  • Move your /var/log to another partition with enough free space (following your example, that would be /home/log)
  • Create a symbolic link from /var/log to /home/log (ln -s /home/log /var/log)
  • Restart the processes that you stopped in the first step

Please note that this is far from what I'd consider as a good practice. It's just a workaround so that you don't have to shutdown the server. The right solution would be to create a new /var or /var/log partition with enough space (or expand the current one),

share|improve this answer
    
Would there be any effect from doing the link idea if the logrotate is run? I didn't know if it would overwrite or screw up locations of files if they weren't already links...that could mean more maintenance headaches later. –  Bart Silverstrim Aug 19 '09 at 12:57
    
Yes, it finally seems that doing this operation while the server is running is not that good. I won't be able then to see which process still use the old /var/log/ Moreover I think that some applications don't easily deal with symbolic link so that could eventually screw up the log. Maybe I should schedule a maintenance mode. Actually I'm manually deleting the files when the left space becomes short. mriedman: I'll check if I can resize that partition –  Razique Aug 20 '09 at 9:25

Everyone else's answers are excellent and correct, and you should definitely read them first.

I just thought I'd share this because it makes for easy copy-and-paste, if your case turns out to be quite a simple one like mine was:

Stop the syslog and copy current logs out:

service rsyslog stop
mkdir -p /tmp/varlog
cp -r /var/log/* /tmp/varlog

then, mount your new location over /var/log. Say it's a new device called /dev/sdb

mount /dev/sdb /var/log

now you can copy files back and restart the syslog:

cp -r /tmp/varlog/* /var/log
rm -rf /tmp/varlog
service rsyslog start

Assuming this all happens quite early on in the life of your machine, rsyslog is likely to be the only daemon running. YMMV!

PS - you'll be wanting to add it to your fstab as well probably. Here's one way of doing that, again assuming a very straightforward mount:

cat /etc/mtab |grep /var/log >>/etc/fstab

(cf http://serverfault.com/a/267610/80606 about catting mtab to fstab)

share|improve this answer
    
Thanks a lot hwjp –  Razique Jun 22 '12 at 9:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.