Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have servers with huge daily size of log files ~ 1GB. I want to do daily logrotate at midnight. The problem is for each rotated file missing first 17-25 minutes after midnight and same file contains 17-25 minutes data for next day.

My question is how to do properly logrotate for each day and data for that day must be from midnight to midnight.

Server has a few services and each service has 100MB-1,5GB daily file as apache2, lighttpd, ftp, etc.

less /var/log/lighttpd/access.log.1

np.domainexample.co.uk 11.22.33.44 - [16/Oct/2012:00:18:20 +0100] "GET /nowplaying/aim_abc.xml HTTP/1.1" 2422 200 "domainexample/1.3 CFNetwork/596.2.3 Darwin/12.2.0 (x86_64) (iMac9%2C1)" "-"
np.domainexample.co.uk 11.22.33.44 - [16/Oct/2012:00:18:20 +0100] "GET /nowplaying/aim_abc.xml HTTP/1.1" 2422 200 "domainexample/1.3 CFNetwork/596.2.3 Darwin/12.2.0 (x86_64) (iMac12%2C1)" "-"
np.domainexample.co.uk 11.22.33.44 - [16/Oct/2012:00:18:20 +0100] "GET /nowplaying/aim_abr.xml HTTP/1.1" 0 304 "hhh%20Radio%20Umee/616 CFNetwork/548.1.4 Darwin/11.0.0" "-"
np.domainexample.co.uk 11.22.33.44 - [16/Oct/2012:00:18:20 +0100] "GET /nowplaying/aim_abr.xml HTTP/1.1" 2433 200 "NobexRadio" "-"
np.domainexample.co.uk 11.22.33.44 - [16/Oct/2012:00:18:20 +0100] "GET /nowplaying/aim_abr.xml HTTP/1.1" 2433 200 "domainexample/1.3 CFNetwork/520.4.3 Darwin/11.4.0 (x86_64) (iMac11%2C2)" "-"

tail /var/log/lighttpd/access.log.1

np.domainexample.co.uk 11.22.33.44 - [17/Oct/2012:00:21:56 +0100] "GET /nowplaying/aim_abr.xml HTTP/1.1" 2121 200 "domainexample/1.3 CFNetwork/520.5.1 Darwin/11.4.2 (x86_64) (iMac12%2C2)" "-"
exam.as5623.net 11.22.33.44 - [17/Oct/2012:00:21:56 +0100] "GET /oconnell/20121012170055.mp3 HTTP/1.1" 78118 206 "AppleCoreMedia/1.0.0.10A403 (iPod; U; CPU OS 6_0 like Mac OS X; en_gb)" "-"
exam.as5623.net 11.22.33.44 - [17/Oct/2012:00:21:56 +0100] "GET /oconnell/20121001174703.mp3 HTTP/1.1" 78120 206 "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "-"
exam.as5623.net 11.22.33.44 - [17/Oct/2012:00:21:56 +0100] "GET /frankskinner/20120905121827.mp3 HTTP/1.1" 78117 206 "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_gb)" "-"
np.domainexample.co.uk 11.22.33.44 - [17/Oct/2012:00:21:56 +0100] "GET /nowplaying/aim_ab8.xml HTTP/1.1" 0 304 "hhh%2080s/616 CFNetwork/609 Darwin/13.0.0" "-"

bzless /var/log/lighttpd/access.log.2.bz2

exam.as5623.net 11.22.33.44 - [15/Oct/2012:00:17:03 +0100] "GET /dave_gorman/20121014130335.mp3 HTTP/1.1" 78117 206 "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "-"
np.domainexample.co.uk 11.22.33.44 - [15/Oct/2012:00:17:03 +0100] "GET /nowplaying/aim_ab8.xml HTTP/1.1" 2190 200 "hhh/1 CFNetwork/520.4.3 Darwin/11.4.0 (x86_64) (iMac12%2C1)" "-"
exam.as5623.net 11.22.33.44 - [15/Oct/2012:00:17:03 +0100] "GET /frankskinner/20121013132200.mp3 HTTP/1.1" 284941 206 "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_gb)" "-"
exam.as5623.net 11.22.33.44 - [15/Oct/2012:00:17:03 +0100] "GET /dave_gorman/20121014130335.mp3 HTTP/1.1" 78117 206 "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "-"
exam.as5623.net 11.22.33.44 - [15/Oct/2012:00:17:03 +0100] "GET /dave_gorman/20121014130335.mp3 HTTP/1.1" 78117 206 "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "-"

bztail /var/log/lighttpd/access.log.2.bz2

np.domainexample.co.uk 11.22.33.44 - [16/Oct/2012:00:18:20 +0100] "GET /nowplaying/aim_abr.xml HTTP/1.1" 2433 200 "hhh/1.3 CFNetwork/454.12.4 Darwin/10.8.0 (i386) (MacBookPro5%2C5)" "-"
exam.as5623.net 11.22.33.44 - [16/Oct/2012:00:18:20 +0100] "GET /dave_gorman/20121014130335.mp3 HTTP/1.1" 41486 206 "AppleCoreMedia/1.0.0.10A403 (iPod; U; CPU OS 6_0 like Mac OS X; en_gb)" "-"
exam.as5623.net 11.22.33.44 - [16/Oct/2012:00:18:20 +0100] "GET /frankskinner/20121013132200.mp3 HTTP/1.1" 78377 206 "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_gb)" "-"
np.domainexample.co.uk 11.22.33.44 - [16/Oct/2012:00:18:20 +0100] "GET /nowplaying/aim_abc.xml HTTP/1.1" 0 304 "Classic%20Rock/616 CFNetwork/609 Darwin/13.0.0" "-"

bzless /var/log/lighttpd/access.log.3.bz2

np.domainexample.co.uk 11.22.33.44 - [14/Oct/2012:00:19:03 +0100] "GET /nowplaying/aim_ab0.xml HTTP/1.1" 0 304 "hhh%2080s/616 CFNetwork/609 Darwin/13.0.0" "-"
exam.as5623.net 11.22.33.44 - [14/Oct/2012:00:19:03 +0100] "GET /oconnell/20120920182425.mp3 HTTP/1.1" 78118 206 "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "-"
np.domainexample.co.uk 11.22.33.44 - [14/Oct/2012:00:19:03 +0100] "GET /nowplaying/aim_ab0.xml HTTP/1.1" 0 304 "hhh%2080s/616 CFNetwork/548.1.4 Darwin/11.0.0" "-"
np.domainexample.co.uk 11.22.33.44 - [14/Oct/2012:00:19:03 +0100] "GET /nowplaying/aim_abc.xml HTTP/1.1" 0 304 "hhh%20Radio%20Umee/616 CFNetwork/609 Darwin/13.0.0" "-"
np.domainexample.co.uk 11.22.33.44 - [14/Oct/2012:00:19:03 +0100] "GET /nowplaying/aim_ab9.xml HTTP/1.1" 0 304 "hhh%20Radio%20Umee/616 CFNetwork/548.1.4 Darwin/11.0.0" "-"

bztail /var/log/lighttpd/access.log.3.bz2

np.domainexample.co.uk 11.22.33.44 - [15/Oct/2012:00:17:02 +0100] "GET /nowplaying/ab7.xml HTTP/1.1" 1295 200 "PHP/5.2.6" "-"
np.domainexample.co.uk 11.22.33.44 - [15/Oct/2012:00:17:02 +0100] "GET /tracks/7869.xml HTTP/1.1" 1998 200 "PHP/5.2.6" "-"
exam.as5623.net 11.22.33.44 - [15/Oct/2012:00:17:02 +0100] "GET /frankskinner/20121013132200.mp3 HTTP/1.1" 230649 206 "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_gb)" "-"
exam.as5623.11.22.33.44 - [15/Oct/2012:00:17:02 +0100] "GET /frankskinner/20121013132200.mp3 HTTP/1.1" 78377 206 "AppleCoreMedia/1.0.0.10A403 (iPad; U; CPU OS 6_0 like Mac OS X; en_us)" "-"
np.domainexample.co.uk 11.22.33.44 - [15/Oct/2012:00:17:02 +0100] "GET /nowplaying/abx.xml HTTP/1.1" 314 200 "PHP/5.2.6" "-"

cat /etc/logrotate.d/lighttpd

/var/log/lighttpd/*.log {
    daily
   missingok
   rotate 999
   compresscmd /bin/bzip2
   compressext .bz2
   compress
   delaycompress
   notifempty
   copytruncate
   sharedscripts
   postrotate
      if [ -f /var/run/lighttpd.pid ]; then \
        if [ -x /usr/sbin/invoke-rc.d ]; then \
           invoke-rc.d lighttpd force-reload > /dev/null; \
        else \
           /etc/init.d/lighttpd force-reload > /dev/null; \
        fi; \
      fi;
   endscript
}

ls /etc/logrotate.d/

apache2
apt
aptitude
dpkg
lighttpd
ocsinventory-agent
ppp
rsyslog.disabled
syslog-ng
vsftpd

Cheers,

E

share|improve this question

migrated from stackoverflow.com Oct 18 '12 at 6:57

This question came from our site for professional and enthusiast programmers.

1 Answer 1

up vote 0 down vote accepted

Do you really need from midnight-to-midnight without any log-entry of the previous/following day?

Then you need to do post-processing of the log-files where you cut'n'paste your older logs to the right borders.

If you could time the run of your logrotation better, you are always running in race conditions between rotating all you logfiles and users accessing your servers at the same time, so for perfect one-day-logfiles you need post processing...

share|improve this answer
    
> Do you really need from midnight-to-midnight without any log-entry of > the previous/following day? Yes We have 999 log rotated files. Problem is only files contains data with offset 15-25 min from midnight current day and after midnight next day. We need daily log rotated file with data from 00:00:00 to 23-59:59 compressed in GZ format. Best, E –  Lihnjo Nov 30 '12 at 14:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.