Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So I switched all my sites over to Amazon's EC2 service a few months ago and I noticed today that my drive capacity is at >90%. Did a little digging and it seems that the majority of that is being used by access.log files created by apache.

I am wondering what the best method is to deal with this. My knee jerk reaction is to write a small script that runs daily [or weekly, monthly, whatever] that will zip all of them and put them someplace else in the event I need them in the future -- most likely Amazon's S3.

Is that the best method for this or is there a standard practice that others are using that I should follow?

EDIT: To answer a commenter, this server is currently running Linux -- Ubuntu to be precise

share|improve this question
    
IS this a Linux based machine? –  Iain Jul 13 '12 at 7:59
    
@Iain - Yes, running on Ubuntu –  Andrew G. Johnson Jul 13 '12 at 8:00
1  
Logrotate is your friend then –  Iain Jul 13 '12 at 8:00
    
@Iain - thanks after Googling it, seems as though logrotate does what I asked but instead of sending to S3 sends to an email [correct me if I'm wrong.] -- my worry there is my gmail inbox getting too full, it sounds as though my S3 script may be my best bet –  Andrew G. Johnson Jul 13 '12 at 8:02
1  
Google more. Look at e.g. thegeekstuff.com/2010/07/logrotate-examples Logrotate does everything you need + ensures that logging is not interrupted. You can execute custom commands after rotating is done, thus you are free to add the appropriate copy and delete commands there. You do not have to send logs by e-mail from logrotate. –  jpe Jul 13 '12 at 8:19

2 Answers 2

Use the logrotate utility to rotate your logs. You can use the postrotate option to run a command to shunt them off to S3 something like

/usr/bin/s3cmd put /var/log/apache2/access.log.1 s3://your.s3.host/access.log.1  

would to the trick. You'll probably want to use the datext option so that your files are dated e.g. access.log-20120713 so your postrotate script becomes slightly more complicated

/usr/bin/s3cmd put /var/log/apache2/access.log-$(date "+%Y%m%d") s3://your.s3.host/access.log-$(date "+%Y%m%d")

If you use the compress option (recommended) your postrotate becomes

/usr/bin/s3cmd put /var/log/apache2/access.log-$(date "+%Y%m%d").gz s3://your.s3.host/access.log-$(date "+%Y%m%d"),gz
share|improve this answer
    
Didn't know about the postrotate option, neat! –  Kenny Rasschaert Jul 13 '12 at 8:24

The first thing you should consider is compression. Because of the way Apache logs are formatted, they compress really nicely.

Like Iain suggested, logrotate is your friend. It can take care of the compression and will also allow you to specify how long to want to retain your logs.

If you insist on keeping all logs, you could set up logrotate to mail you old logs, or set up a secondary script with cron to store the old logs on S3 or elsewhere.

Here's a good resource to get you started on using logrotate if you've never used it before.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.