How can I move /var/log directory

Question

on our production server, there is a small drive for the main mount point "/" /var/log is taking much space and I have to manually delete some files. How can I move /var/log/ to let's say /home/log WITHOUT REBOOTING.

Here is the thing I thought :

$mkdir /home/log

$rsync -a /var/log /home/log

$mount --bind /home/log /var/log

$/etc/init.d/rsyslof/restart

BUT : I know that some services use file descriptors, so they'll continue to use /var/log or inodes. Thanks

Answer 1

Proper design

I assume you are unable to simply extend the filesystem in question (using lvextend && ext2online), because you do not use LVM or use wrong filesystem type.

Your approach

What you've proposed might work if you signal the daemons with SIGHUP (kill -1 pid). Obviously you would need to later on "mount -o bind / /somewhere" and clean up what has been left underneath mounted /var/log. But it has a bad smell for me, especially for production.

Avoid downtime, have a clean result (but complicated to do)

Forget about "mount -o bind" idea, create a new LV/partition, but don't mount it yet.

lsof | grep /var/log             # lists open files in /var/log

For each daemon that has any open file (I would expect at least syslog, inetd, sshd):

• reconfigure the daemon no to log to /var/log
• refresh the daemon (kill -1 or /etc/init.d/script reload)
• confirm with lsof | grep /var/log that daemon has closed its files

Mount over /var/log. Restore old configurations, SIGHUP/reload daemons again.

Easy way (downtime)

Create a new LV/partition and mount it properly over either /var or /var/log. The easy way is to take down the server to maintenance mode (single-user mode), and use the actual console (not ssh) for the operation.

Answer 2

Another thing that you could do is:

• Stop the processes that have open files on /var/log
• Verify that there aren't any processes with open files on /var/log (using lsof as kubanskamac suggested)
• Move your /var/log to another partition with enough free space (following your example, that would be /home/log)
• Create a symbolic link from /var/log to /home/log (ln -s /home/log /var/log)
• Restart the processes that you stopped in the first step

Please note that this is far from what I'd consider as a good practice. It's just a workaround so that you don't have to shutdown the server. The right solution would be to create a new /var or /var/log partition with enough space (or expand the current one),