Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've been told that our (scripted) daily reboot of apache has been set up because of our log rotation. The concern is that logrotate might lose a line in the access logs if it rotates while apache is running and accepting requests.

Can someone tell me if this is plausible, true, or complete nonsense?

share|improve this question
1  
Is apache actually being rebooted or just SIGHUP'd? –  Sean Bright Aug 7 '09 at 19:19
add comment

migrated from stackoverflow.com Aug 7 '09 at 19:34

This question came from our site for professional and enthusiast programmers.

4 Answers

up vote 8 down vote accepted

You need to restart the server after the logs have been rotated as Apache holds open the file handle to the log and will continue writing to the old log file if you don't. Sending USR1 to the parent (graceful restart) should do the trick.

Reference: 1.3, 2.2

Quote from 2.2 docs:

On even a moderately busy server, the quantity of information stored in the log files is very large. The access log file typically grows 1 MB or more per 10,000 requests. It will consequently be necessary to periodically rotate the log files by moving or deleting the existing logs. This cannot be done while the server is running, because Apache will continue writing to the old log file as long as it holds the file open. Instead, the server must be restarted after the log files are moved or deleted so that it will open new log files.

By using a graceful restart, the server can be instructed to open new log files without losing any existing or pending connections from clients. However, in order to accomplish this, the server must continue to write to the old log files while it finishes serving old requests. It is therefore necessary to wait for some time after the restart before doing any processing on the log files. A typical scenario that simply rotates the logs and compresses the old logs to save space is:

mv access_log access_log.old
mv error_log error_log.old
apachectl graceful sleep 600
gzip access_log.old error_log.old

Another way to perform log rotation is using piped logs as discussed in the next section.

share|improve this answer
    
@tvanfosson, we've seen lots of problems where httpd's don't shutdown gracefully to the point where we currently have to HUP the server nightly as a part of our log rotation effort. –  Rob Wells Aug 7 '09 at 19:43
add comment

Totally plausible. Depends on how you are stopping Apache.

If you are doing a nightly HUP to get the servers to start the new log then you should not have a problem. Any clumsy stop and restart and your own your own though!

However, any connections present during the nightly HUP will be lost and those punters will have to reconnect. Bit of a pity if they're in the middle of a big download!

If you're logging the bytes served as a part of the request, then the connection isn't logged until the download is complete, hence those connections that were dropped because of the HUP will be missing from the logs.

Have you looked at the new 2.2.12 release?

One of the changes is to make the "piped through rotatelogs" more robust so that you can defer the rotation out to the rotatelogs utility and let it top and tail your logs without the need to perform the nightly HUP. So no dropped connections and no missing log transactions if your logging bytes served.

The doc's say

The rotation occurs at the beginning of this interval. For example, if the rotation time is 3600, the log file will be rotated at the beginning of every hour; if the rotation time is 86400, the log file will be rotated every night at midnight.

If you look inside rotatelogs it's not obvious how they're doing this as there's no magic numbers at all until you spot the line:

tLogStart = (now / tRotation) * tRotation;

And this will align the start time with the start of the current hour (for tRotation == 3600) or the previous midnight (for tRotation == 86400). Who knows what it'll do for any other numbers, e.g. 16432?

Edit: I forgot to say that we've seen problems with graceful not behaving very gracefully. Specifically, just leaving child processes hanging. This seems to have been acknowledged as v2.2.12 now has a new config directive GracefulShutdownTimeout which specifies a timeout after which the httpd process terminates irrespective of any requests being served.

HTH

'Avahappy,

share|improve this answer
add comment

Why is loosing one line such a big deal? If its that important, perhaps the logs need to be printed to green bar as a backup.

share|improve this answer
1  
@Justin, it could be a big deal if you have auditing and archiving requirements that you have to legally comply with. –  Rob Wells Aug 7 '09 at 19:42
    
True. But I still think green bar is a nice idea. +1 insightful. –  Kyle Hodgson Aug 7 '09 at 21:54
    
@Rob that is true. However, I'm told some really big sites (like slashdot) don't log because its a storage and performance killer. Of course I would not stop logging unless I absolutely had to considering how well apache logs compress. –  Justin Dearing Aug 8 '09 at 1:28
    
@Rob, I think its a valid question to ask why a line matters, if it does then simple logging shouldn't be used. It should be captured & secured better since since the logs could still be manually manipulated. –  SpaceManSpiff Aug 8 '09 at 3:02
    
@SpaceManSpiff, you are totally correct. It needs to be captured and secured in a better way. @Justin, you are also totally correct in that it is a storage and performance killer. Both of which we are trying to address at the moment for one of the major web sites in the world that I happen to be involved with. Possibly using centralised logging. BTW The web site involved has 400,000+ updates per day and peak hit rates of millions of hits per second for more than an hour! –  Rob Wells Aug 10 '09 at 9:47
add comment

Yep, as per the other answers, the issue is that apache will continue to write to the file handle it has open.

We (and lots of other people) use cronlog to avoid this. In this setup apache pipes the logs to cronolog, and the server does not need to be restarted/reloaded.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.