What do you guys use to manage your log size? How long do you keep logs archived? I realize this ultimately depends upon site traffic and the server hardware. I have 4GB of RAM on my server with 2x dual core 2.0 GHz processors running Ubuntu Server. I don't have any huge sites running on it just yet, I want to get all these details worked out before I transfer them.


I use logrotate to do daily rotation (with compression for big logs), and tweak how many days I keep based on the content of the logs and how long I need it (and how big they get!).

You could also dump all your logs into a central Splunk server and delete them. Of course, then it's just a matter of managing the disk usage in the Splunk server...

  • I like that idea... I'll test it out this evening. If I have a problem during boot - where would I find the log file for that?
    – Webnet
    Aug 23 '10 at 17:29
  • It'll be in your cron logs, wherever those are.
    – Bill Weiss
    Aug 23 '10 at 17:37
  • That's for the logrotate answer. Splunk, well, they would be in the Splunk logs :)
    – Bill Weiss
    Aug 23 '10 at 17:38
  • If I'm not mistaken, isn't splunk mainly just for viewing the logs? If I configure /home/*.log will that apply to all log files within that directory? I've moved error_log's into those directories as well, do I need to configure each seperately?
    – Webnet
    Aug 23 '10 at 23:12
  • It can take wildcards for what files to read. As well, it slurps all that stuff up into its index, so you could delete the files once you verify they're in there.
    – Bill Weiss
    Aug 25 '10 at 20:39

It depends on you - which log rotation method are you using on the servers the sites are currently on? How long do you think you'll need logs for? How much traffic will the sites be taking? The resources you've posted are largely irrelevant, the RAM/CPU will only really come into play whilst writing the logs (which you're not going to change), analysing them and compressing them - the amount of free disk space is more relevant.

I'd recommend compressing them after each day (so log.1.gz log.2.gz etc.) and deleting any over a week old.


I use something like Bill, but because i use the free license i can index only 500 MB / day. So for my case it's bad to dump everything in splunk. Instead i save them localy and index them on demand (For troubleshooting purposes, i need 4-5 logs as an average)

How many logs do you have per day ? And what's the possibility to need them ? If you have a lot of space, i suggest to keep the logs uncompressed for 2,3 days, and afterwards compress them.

Also depending on your traffic, addinionaly to daily rotate, it would be good to rotate them if the exceed some threshold. Let's say 500 MB. It's rather impossible to work with a 5 GB file full of logs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.