What is the best solution for log archiving ? I have two application server (tomcat6) and I need to keep application log files for 7 years. Shipping, zipping and writing them on Bluray or DVD is time consuming, some the process is already automated but still takes so long, what do you suggest ?

  • How large are the logs? How much data is being added per log rotation? The file size and rotation scheduled are key to finding a good solution. – jeffatrackaid Jan 2 '14 at 17:15

The simpliest solution to me is to use syslog or one of its derivatives (rsyslog, syslog-ng) to send all logs to central servers. (Plural because you may want to have some kind of failover for security, or load-balancing if the log volume is very high)

Primary components you need are:

  • Compress logs with the most efficient algorithm you can afford depending on the log volume and computing power on the central server(s). gzip with the default aggressivity setting should do the job.
  • Use the most resilient storage solution you can affrod (RAID5 seems a minimum)

Log files are very repetitive, so a day worth of application logs should have a very high compression ratio, allowing you to store your 7 years in a decent amount of hard-drive.

  • We are doing something similar and the mzsql built-in myisam pack is enough as today drives are big enough and you can keep a lot of data online and available without big costs. – Jure1873 Jan 2 '14 at 16:04
  • Indeed. You can store a hell of a lot of log lines in a terByte-big HDD. Even Enterprise-grade drives aren't that expensive, even considering a fully-redondant storage solution. – mveroone Jan 2 '14 at 16:06
  • 1
    +1. I have been gzipping my logfiles for years. Gzip's default encryption provides a decent balance between CPU usage, the time required to compress the data and file size. 100GB of logs is typically compressed down to 10GB. Higher compression methods like bzip and gzip --best use more CPU without significant reduction in file size. Another option is to use a filesystem like ZFS or a storage appliance like FreeNAS and use deduplication and/or compression together; and then optionally back up to tape afterwards. These solutions will typically provide RAID for disk redundancy. – Stefan Lasiewski Jan 2 '14 at 16:09
  • Too bad this benchmark doesn't specify compress time. Intresting whatsoever ! – mveroone Jan 3 '14 at 8:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.