What do you guys use to manage your log size? How long do you keep logs archived? I realize this ultimately depends upon site traffic and the server hardware. I have 4GB of RAM on my server with 2x dual core 2.0 GHz processors running Ubuntu Server. I don't have any huge sites running on it just yet, I want to get all these details worked out before I transfer them.
logrotate to do daily rotation (with compression for big logs), and tweak how many days I keep based on the content of the logs and how long I need it (and how big they get!).
You could also dump all your logs into a central Splunk server and delete them. Of course, then it's just a matter of managing the disk usage in the Splunk server...
It depends on you - which log rotation method are you using on the servers the sites are currently on? How long do you think you'll need logs for? How much traffic will the sites be taking? The resources you've posted are largely irrelevant, the RAM/CPU will only really come into play whilst writing the logs (which you're not going to change), analysing them and compressing them - the amount of free disk space is more relevant.
I'd recommend compressing them after each day (so log.1.gz log.2.gz etc.) and deleting any over a week old.
I use something like Bill, but because i use the free license i can index only 500 MB / day. So for my case it's bad to dump everything in splunk. Instead i save them localy and index them on demand (For troubleshooting purposes, i need 4-5 logs as an average)
How many logs do you have per day ? And what's the possibility to need them ? If you have a lot of space, i suggest to keep the logs uncompressed for 2,3 days, and afterwards compress them.
Also depending on your traffic, addinionaly to daily rotate, it would be good to rotate them if the exceed some threshold. Let's say 500 MB. It's rather impossible to work with a 5 GB file full of logs.