What is the best solution for log archiving ? I have two application server (tomcat6) and I need to keep application log files for 7 years. Shipping, zipping and writing them on Bluray or DVD is time consuming, some the process is already automated but still takes so long, what do you suggest ?
The simpliest solution to me is to use
syslog or one of its derivatives (rsyslog, syslog-ng) to send all logs to central servers. (Plural because you may want to have some kind of failover for security, or load-balancing if the log volume is very high)
Primary components you need are:
- Compress logs with the most efficient algorithm you can afford depending on the log volume and computing power on the central server(s).
gzipwith the default aggressivity setting should do the job.
- Use the most resilient storage solution you can affrod (RAID5 seems a minimum)
Log files are very repetitive, so a day worth of application logs should have a very high compression ratio, allowing you to store your 7 years in a decent amount of hard-drive.