19

On our production server there is a small drive for the root mount point /, /var/log is taking too much space and I have to manually delete some files. How can I move /var/log/ to let's say /home/log WITHOUT REBOOTING?

Here is the thing I thought:

$ mkdir /home/log
$ rsync -a /var/log /home/log
$ mount --bind /home/log /var/log
$ /etc/init.d/rsyslog restart

But I know that some services use file descriptors, so they'll continue to use /var/log or inodes.

2
  • There's a detailed answer to essentially the same question over at AskUbuntu at askubuntu.com/a/346579/422224
    – Eborbob
    Mar 19, 2017 at 18:57
  • This is not an answer but a suggestion. if possible use for logs a file system with realtime compresion , like btrfs. When you mount this partition, use the mount option "compress=lzo" or "compress=zlib". lzo offers better performance, zlib better compression.
    – Massimo
    Jun 15, 2017 at 6:32

6 Answers 6

27

Proper design

I assume you are unable to simply extend the filesystem in question (using lvextend && ext2online), because you do not use LVM or use wrong filesystem type.

Your approach

What you've proposed might work if you signal the daemons with SIGHUP (kill -1 pid). Obviously you would need to later on "mount -o bind / /somewhere" and clean up what has been left underneath mounted /var/log. But it has a bad smell for me, especially for production.

Avoid downtime, have a clean result (but complicated to do)

Forget about "mount -o bind" idea, create a new LV/partition, but don't mount it yet.

lsof | grep /var/log             # lists open files in /var/log

For each daemon that has any open file (I would expect at least syslog, inetd, sshd):

  • reconfigure the daemon no to log to /var/log
  • refresh the daemon (kill -1 or /etc/init.d/script reload)
  • confirm with lsof | grep /var/log that daemon has closed its files

Mount over /var/log. Restore old configurations, SIGHUP/reload daemons again.

Easy way (downtime)

Create a new LV/partition and mount it properly over either /var or /var/log. The easy way is to take down the server to maintenance mode (single-user mode), and use the actual console (not ssh) for the operation.

1
  • how to be sure nothing will be lost during the laps between startup and /var/log mounted?
    – Sandburg
    Apr 4, 2020 at 14:37
10

Everyone else's answers are excellent and correct, and you should definitely read them first.

I just thought I'd share this because it makes for easy copy-and-paste, if your case turns out to be quite a simple one like mine was:

Stop the syslog and copy current logs out:

service rsyslog stop
mkdir -p /tmp/varlog
cp -r /var/log/* /tmp/varlog

then, mount your new location over /var/log. Say it's a new device called /dev/sdb

mount /dev/sdb /var/log

now you can copy files back and restart the syslog:

cp -r /tmp/varlog/* /var/log
rm -rf /tmp/varlog
service rsyslog start

Assuming this all happens quite early on in the life of your machine, rsyslog is likely to be the only daemon running. YMMV!

PS - you'll be wanting to add it to your fstab as well probably. Here's one way of doing that, again assuming a very straightforward mount:

cat /etc/mtab |grep /var/log >>/etc/fstab

(cf https://serverfault.com/a/267610/80606 about catting mtab to fstab)

3
  • 7
    Nice solution. The one thing I'd change is replacing your cp -r with sudo tar cpf and sudo tar xpf to preserve file ownership and permissions on the off chance there are logs created by a user other than root. Aug 22, 2015 at 14:21
  • We're copying it to a new location /tmp/varlog. Means we have 2 copies now. How do we delete the old log files after the mounting process? I presume we can no longer access the old /varlog. Should we do a move instead of copy?
    – tinker
    Oct 12, 2020 at 9:43
  • I was thinking rsync -qaAHX.
    – TonyG
    Sep 2, 2022 at 21:33
2

Another thing that you could do is:

  • Stop the processes that have open files on /var/log
  • Verify that there aren't any processes with open files on /var/log (using lsof as kubanskamac suggested)
  • Move your /var/log to another partition with enough free space (following your example, that would be /home/log)
  • Create a symbolic link from /var/log to /home/log (ln -s /home/log /var/log)
  • Restart the processes that you stopped in the first step

Please note that this is far from what I'd consider as a good practice. It's just a workaround so that you don't have to shutdown the server. The right solution would be to create a new /var or /var/log partition with enough space (or expand the current one),

2
  • Would there be any effect from doing the link idea if the logrotate is run? I didn't know if it would overwrite or screw up locations of files if they weren't already links...that could mean more maintenance headaches later. Aug 19, 2009 at 12:57
  • Yes, it finally seems that doing this operation while the server is running is not that good. I won't be able then to see which process still use the old /var/log/ Moreover I think that some applications don't easily deal with symbolic link so that could eventually screw up the log. Maybe I should schedule a maintenance mode. Actually I'm manually deleting the files when the left space becomes short. mriedman: I'll check if I can resize that partition
    – Razique
    Aug 20, 2009 at 9:25
1

To move the log directory to a different location:

sudo service rsyslog stop
cd /var
sudo mv log /path/to/new/location/
sudo ln -s /path/to/new/location/log
sudo service rsyslog start
0

Another solution based on @hwjp , if you can't use another drive volume to move them, you can create a virtual drive volume into another volume who has more space available (that is my case) :

Create virtual volume:

A) do : sudo dd if=/dev/zero of=VHD-log.img bs=1M count=1200

  • if=/dev/zero: input file to provide a character stream for initialising data storage
  • of=VHD.img: image file to be created as storage volume on /thevolumeofyourchoice
  • bs=1M: read and write up to 1M at a time
  • count=1200: copies only 1200M (1GB) input blocks

B) do : sudo mkfs -t ext4 /thevolumeofyourchoice/VHD-log.img Format the EXT4 file system type in the VHD-log image file with the mkfs utility.

C) do : sudo mkdir /thevolumeofyourchoice/vlog Mount VHD-log to a directory (mount point)

D) do : sudo mount -t auto -o loop /thevolumeofyourchoice/VHD-log.img /thevolumeofyourchoice/vlog

D1) To mount VHD-log at system boot to the final directory, add this entry in the /etc/fstab file.

/thevolumeofyourchoice/VHD-log.img  /var/log/  ext4    defaults        0  0

Move old log files:

E) do :

  • service rsyslog stop
  • lsof | grep /var/log to lists open files in /var/log and switch off needed daemon (apach2,freshclam in my case)
  • cp -rp /var/log/* /thevolumeofyourchoice/vlog (cp -p same as --preserve=mode,ownership,timestamps)

F) do :

  • sudo umount /thevolumeofyourchoice/vlog
  • sudo mv /var/log /var/log-old
  • sudo mkdir /var/log
  • sudo chgrp syslog /var/log
  • sudo mount -t auto -o loop /thevolumeofyourchoice/VHD-log.img /var/log

G) do : service rsyslog start and restart others services you stopped

Finally double check :

You ca do a - lsof | grep /var/log to lists open files in /var/log and verify they point to /var/log and not /var/log-old

You can mv, backup or delete /var/log-old after all is ok.

-1

I would just:

apt install lsyncd
mv /var/log /varlog.root
mkdir /var/log /home/log
mount --bind /home/log /var/log

# set up lsyncd appropriately to sync real-time between /varlog.root & /var/log
service lsyncd start
sleep 3m

service rsyslog restart
# Proceed with all the other daemons
# to switch them over to the new /var/log and release /varlog.root
sleep 3m

# Then do this a few times until no processes use the old dir
lsof +D /varlog.root && rm -rf /varlog.root
3
  • Is it good to leave /var/log mounted over /home/log as is? This entire question is not for seasoned admins, I presume.
    – kubanczyk
    May 14, 2018 at 19:19
  • What do you mean? /home/log is mounted over /var/log, not the other way around. And yes, it is not a problem to leave it as is, just make sure it is restored on each boot automatically and early enough for all the processes not to attempt to create some new logs in the empty /var/log mount point. May 15, 2018 at 0:22
  • a lot of times security requirements need /var/log and /var/log/audit on actual separate mount points. Otherwise leave them be and just setup appropriate log rotation to avoid unnecessary disk usage.
    – Brad
    Dec 27, 2018 at 17:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .