0

I have a CentOS box running rsyslog and logrotate as my syslog server for a whole bunch of network devices. I've been toying around with this for a while, the logrotate/compression piece is working ok, but I can't seem to get it to delete the old compressed .gz logs. Here's the basic setup:

Logs are stored in: /var/log/syslog and each host creates a new sub-folder based on its IP address or hostname. I have two particular hosts that are really chatty, so I have specific rules for them to truncate after 5 GB. The idea is to compress and start a new log every day, and keep 60 days' worth of logs.

Here are my configs (most of these were not created by me, but I have maintained and modified several of them):

cat /etc/cron.hourly/logrotate

#!/bin/sh`enter code here`
/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

cat /etc/logrotate.d/syslog

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    compress
    daily
    #delaycompress
    dateext
    missingok
    rotate 60
    maxage 60
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

cat /etc/logrotate.conf

/var/log/syslog/*/*.log {
daily
rotate 60
maxage 60
compress
}

daily
rotate 60
create
dateext
compress
include /etc/logrotate.d

/var/log/wtmp {
    monthly
    create 0664 root utmp
        minsize 1M
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.
# Specific rule for HOST_A
/var/log/syslog/HOST_A/*.log {
daily
size 5G
rotate 30
maxage 30
compress
}

# Specific rule for HOST_B
/var/log/syslog/HOST_B/*.log {
daily
size 5G
rotate 30
maxage 30
compress
}

And here is an example of what I have in my \var\log\syslog\host_a folder:

ls /var/log/syslog/HOST_A/ | wc -l

104

ls /var/log/syslog/HOST_A/ -lh

-rw------- 1 root root  416M Jun 20 23:59 HOST_A_2018_06_20.log.1.gz
-rw------- 1 root root   64M Jun 20 16:18 HOST_A_2018_06_20.log.2.gz
-rw------- 1 root root  1.5G Jun 21 23:59 HOST_A_2018_06_21.log.1.gz
<many files redacted>
-rw------- 1 root root  1.6G Sep  4 23:59 HOST_A_2018_09_04.log.1.gz
-rw------- 1 root root  1.5G Sep  5 23:59 HOST_A_2018_09_05.log.1.gz
-rw------- 1 root root  7.7G Sep  6 10:58 HOST_A_2018_09_06.log

So, you can see above, it's not removing the old archived .gz logs after 60 days. It's keeping them indefinitely. I've had to go in periodically and manually delete older files to keep size down. These files go from 6/20 to 9/6 (today), which is 104 days. I'm sure I have it misconfigured, but not sure quite what I'm missing. Any help would be appreciated.

  • You might want to investigate the output of logrotate -d /etc/logrotate.conf. – Thomas Sep 6 '18 at 16:49
  • Thanks, I actually have run that, and on every folder in my syslog folder, it basically says "log does not need rotating" because it hasn't been 1 day yet. – Jake Sep 6 '18 at 20:26
1

You're using two mechanisms for daiy logfiles, and they're not cooperating with each other.

On the one hand you let syslog generate daily files, on the other hand you let logrotate rotate files. Logrotate will treat each daily file as a separate unique set of files (it doesn't consider HOST_A_2018_09_05.log to be related to HOST_A_2018_09_06.log) to rotate, so it never gets to five files files per pattern.

It's better to either not include the date in the filename, or let syslog do its own pruning of old files.

  • Thanks for the insight. I was wondering if that was might be the case. So, the filename essentially has to be the same, with a numerical iteration, (e.g *.1.log, *.2.log, etc.)? I want to be able to keep compressing them .gz files, so they don't take up so much space. And keeping the dates in the names is also useful when looking back through the logs. It seems like there should be a way to do that and have logrotate prune the old ones and still compress and rename the files. Should I try to find a way in rsyslog to prune them? Create a cron.daily job to delete files older 60 days? – Jake Sep 6 '18 at 20:30
  • If you want to use logrotate, the file generated by syslog has to have the same name. Logrotate can append dates to filenames it rotates. So you log to HOST_A.log and when rotating, logrotate can compress and rename it to HOST_A.20180906.log.gz, see the dateext and datefmt parameters for logrotate. – Dennis Kaarsemaker Sep 6 '18 at 20:39
  • This is the section in my /etc/rsyslog.conf file that writes the files: $template RemoteHost,"/var/log/syslog/%HOSTNAME%/%HOSTNAME%_%$YEAR%_%$MONTH%_%$DAY%.log" So, I could change that, then add the dateext and datefmt options to the logrotate settings to fix the problem? – Jake Sep 6 '18 at 20:54
  • I changed my /etc/rsyslog.conf file to no longer add dates. I changed it to: $template RemoteHost,"/var/log/syslog/%HOSTNAME%/%HOSTNAME%.log" Then I went into my /etc/logrotate.conf and added parameters to add the date to my rotated log files, like this: dateext, dateyesterday, dateformat-%Y-%m-%d. That seems to be working pretty well. I'm still waiting for the 45 day mark for some of my older logs to start getting pruned. – Jake Oct 8 '18 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.