14

On our production server there is a small drive for the root mount point /, /var/log is taking too much space and I have to manually delete some files. How can I move /var/log/ to let's say /home/log WITHOUT REBOOTING?

Here is the thing I thought:

$ mkdir /home/log
$ rsync -a /var/log /home/log
$ mount --bind /home/log /var/log
$ /etc/init.d/rsyslof restart

But I know that some services use file descriptors, so they'll continue to use /var/log or inodes.

  • There's a detailed answer to essentially the same question over at AskUbuntu at askubuntu.com/a/346579/422224 – Eborbob Mar 19 '17 at 18:57
  • This is not an answer but a suggestion. if possible use for logs a file system with realtime compresion , like btrfs. When you mount this partition, use the mount option "compress=lzo" or "compress=zlib". lzo offers better performance, zlib better compression. – Massimo Jun 15 '17 at 6:32
23

Proper design

I assume you are unable to simply extend the filesystem in question (using lvextend && ext2online), because you do not use LVM or use wrong filesystem type.

Your approach

What you've proposed might work if you signal the daemons with SIGHUP (kill -1 pid). Obviously you would need to later on "mount -o bind / /somewhere" and clean up what has been left underneath mounted /var/log. But it has a bad smell for me, especially for production.

Avoid downtime, have a clean result (but complicated to do)

Forget about "mount -o bind" idea, create a new LV/partition, but don't mount it yet.

lsof | grep /var/log             # lists open files in /var/log

For each daemon that has any open file (I would expect at least syslog, inetd, sshd):

  • reconfigure the daemon no to log to /var/log
  • refresh the daemon (kill -1 or /etc/init.d/script reload)
  • confirm with lsof | grep /var/log that daemon has closed its files

Mount over /var/log. Restore old configurations, SIGHUP/reload daemons again.

Easy way (downtime)

Create a new LV/partition and mount it properly over either /var or /var/log. The easy way is to take down the server to maintenance mode (single-user mode), and use the actual console (not ssh) for the operation.

8

Everyone else's answers are excellent and correct, and you should definitely read them first.

I just thought I'd share this because it makes for easy copy-and-paste, if your case turns out to be quite a simple one like mine was:

Stop the syslog and copy current logs out:

service rsyslog stop
mkdir -p /tmp/varlog
cp -r /var/log/* /tmp/varlog

then, mount your new location over /var/log. Say it's a new device called /dev/sdb

mount /dev/sdb /var/log

now you can copy files back and restart the syslog:

cp -r /tmp/varlog/* /var/log
rm -rf /tmp/varlog
service rsyslog start

Assuming this all happens quite early on in the life of your machine, rsyslog is likely to be the only daemon running. YMMV!

PS - you'll be wanting to add it to your fstab as well probably. Here's one way of doing that, again assuming a very straightforward mount:

cat /etc/mtab |grep /var/log >>/etc/fstab

(cf https://serverfault.com/a/267610/80606 about catting mtab to fstab)

  • 5
    Nice solution. The one thing I'd change is replacing your cp -r with sudo tar cpf and sudo tar xpf to preserve file ownership and permissions on the off chance there are logs created by a user other than root. – Brenda Bell Aug 22 '15 at 14:21
2

Another thing that you could do is:

  • Stop the processes that have open files on /var/log
  • Verify that there aren't any processes with open files on /var/log (using lsof as kubanskamac suggested)
  • Move your /var/log to another partition with enough free space (following your example, that would be /home/log)
  • Create a symbolic link from /var/log to /home/log (ln -s /home/log /var/log)
  • Restart the processes that you stopped in the first step

Please note that this is far from what I'd consider as a good practice. It's just a workaround so that you don't have to shutdown the server. The right solution would be to create a new /var or /var/log partition with enough space (or expand the current one),

  • Would there be any effect from doing the link idea if the logrotate is run? I didn't know if it would overwrite or screw up locations of files if they weren't already links...that could mean more maintenance headaches later. – Bart Silverstrim Aug 19 '09 at 12:57
  • Yes, it finally seems that doing this operation while the server is running is not that good. I won't be able then to see which process still use the old /var/log/ Moreover I think that some applications don't easily deal with symbolic link so that could eventually screw up the log. Maybe I should schedule a maintenance mode. Actually I'm manually deleting the files when the left space becomes short. mriedman: I'll check if I can resize that partition – Razique Aug 20 '09 at 9:25
-1

I would just:

apt install lsyncd
mv /var/log /varlog.root
mkdir /var/log /home/log
mount --bind /home/log /var/log

# set up lsyncd appropriately to sync real-time between /varlog.root & /var/log
service lsyncd start
sleep 3m

service rsyslog restart
# Proceed with all the other daemons
# to switch them over to the new /var/log and release /varlog.root
sleep 3m

# Then do this a few times until no processes use the old dir
lsof +D /varlog.root && rm -rf /varlog.root
  • Is it good to leave /var/log mounted over /home/log as is? This entire question is not for seasoned admins, I presume. – kubanczyk May 14 '18 at 19:19
  • What do you mean? /home/log is mounted over /var/log, not the other way around. And yes, it is not a problem to leave it as is, just make sure it is restored on each boot automatically and early enough for all the processes not to attempt to create some new logs in the empty /var/log mount point. – Alexander Shcheblikin May 15 '18 at 0:22
  • a lot of times security requirements need /var/log and /var/log/audit on actual separate mount points. Otherwise leave them be and just setup appropriate log rotation to avoid unnecessary disk usage. – Brad Dec 27 '18 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.