0

So I switched all my sites over to Amazon's EC2 service a few months ago and I noticed today that my drive capacity is at >90%. Did a little digging and it seems that the majority of that is being used by access.log files created by apache.

I am wondering what the best method is to deal with this. My knee jerk reaction is to write a small script that runs daily [or weekly, monthly, whatever] that will zip all of them and put them someplace else in the event I need them in the future -- most likely Amazon's S3.

Is that the best method for this or is there a standard practice that others are using that I should follow?

EDIT: To answer a commenter, this server is currently running Linux -- Ubuntu to be precise

5
  • IS this a Linux based machine?
    – user9517
    Jul 13, 2012 at 7:59
  • @Iain - Yes, running on Ubuntu Jul 13, 2012 at 8:00
  • 1
    Logrotate is your friend then
    – user9517
    Jul 13, 2012 at 8:00
  • @Iain - thanks after Googling it, seems as though logrotate does what I asked but instead of sending to S3 sends to an email [correct me if I'm wrong.] -- my worry there is my gmail inbox getting too full, it sounds as though my S3 script may be my best bet Jul 13, 2012 at 8:02
  • 1
    Google more. Look at e.g. thegeekstuff.com/2010/07/logrotate-examples Logrotate does everything you need + ensures that logging is not interrupted. You can execute custom commands after rotating is done, thus you are free to add the appropriate copy and delete commands there. You do not have to send logs by e-mail from logrotate.
    – jpe
    Jul 13, 2012 at 8:19

2 Answers 2

4

Use the logrotate utility to rotate your logs. You can use the postrotate option to run a command to shunt them off to S3 something like

/usr/bin/s3cmd put /var/log/apache2/access.log.1 s3://your.s3.host/access.log.1  

would to the trick. You'll probably want to use the datext option so that your files are dated e.g. access.log-20120713 so your postrotate script becomes slightly more complicated

/usr/bin/s3cmd put /var/log/apache2/access.log-$(date "+%Y%m%d") s3://your.s3.host/access.log-$(date "+%Y%m%d")

If you use the compress option (recommended) your postrotate becomes

/usr/bin/s3cmd put /var/log/apache2/access.log-$(date "+%Y%m%d").gz s3://your.s3.host/access.log-$(date "+%Y%m%d"),gz
1
  • Didn't know about the postrotate option, neat! Jul 13, 2012 at 8:24
1

The first thing you should consider is compression. Because of the way Apache logs are formatted, they compress really nicely.

Like Iain suggested, logrotate is your friend. It can take care of the compression and will also allow you to specify how long to want to retain your logs.

If you insist on keeping all logs, you could set up logrotate to mail you old logs, or set up a secondary script with cron to store the old logs on S3 or elsewhere.

Here's a good resource to get you started on using logrotate if you've never used it before.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .