the main requirement I have is to be able to view and do simple searches on logs combined from multiple machines. However, I would like the solution to have minimal impact on the rest of the (core) system. I don't have any real-time requirements, the process can be asynchronous.
Initially syslog seemed like a good option, but what if the syslog server dies? In the worst case the core system's user see errors, in the best case some logs are lost.
So I started looking around and found Logstash (http://logstash.net/). Currently my idea is:
- on each server (that's running the system's core components) there is a Logstash agent running
- the agent monitores log files and sends them to an ElasticSearch cluster
- there's another server with the Logstash UI
- there's no single point of failure
- even if the ES cluster dies, only the agents will be affected - the application will still happily write logs to files
- after ES comes back, the agent will (hopefully) catch up and send all the pending logs (is Logstash smart enough to do that?)
Do you think this will work? Or maybe you could recommend another solution?