Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am looking into building a syslog / logging infrastructure and am pondering about some architecture best practices. Essentially, I see that a syslog system needs to support two conflicting workloads:

  • log collection. Potentially massive streams of data need to be written quickly to disks and indexed.
  • log querying. logs will be queried by both fixed fields such as date and source as well as text search.

What is the best disk/system setup assuming I'd like to keep it to a single server for now? Should I use SSDs or ramdisk to off-load some processing? some disks in stripe and some in raid5?

I am particularly eyeing Graylog2 with ElasticSearch/MongoDB

share|improve this question
    
Update: the expected log volume is 1-4 gigs/day –  Konrads Sep 11 '12 at 9:09

1 Answer 1

First off I think it helps to define the value of the logs, e.g. If this is high volume financial transaction logs you may opt for very high end raid controllers with lots of battery backed cache and high end disks with tagged writes or NCQ.

In the general case the ZFS filesystem is pretty helpful, you are free to use HDDs for the cheap capacity they offer and then add SSDs as a cache for reading (the L2 ARC cache in ZFS) once you need them. If writes become a bottleneck you can use SSDs for the ZIL (effectively a write cache in ZFS). The good thing is it all just works, really well in my experience too.

Taking it further to address the conflicting workload concerns, a product like Cassandra (there are many other options too) has an architecture Which neatly solves these requirements in an efficient way.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.