We have several (20+) application servers spread across multiple data-centres. We need to centralise the logfiles and monitor them from a single box.
- Large logfiles, in the order of 5-10 Gb per day, per application - so there could be several thousand lines a second.
- Latency is important - we need to be able to react to log events within seconds, if possible.
- Performance footprint should be as low as possible, and should scale predictably with logfile size.
I'd like to get opinions on the best approach to centralise these log files?
One approach we though of was to use Logstash (http://logstash.net/) and Graylog2 (http://graylog2.org/), and send the log events over the network to the monitoring box, either straight TCP, or via a bus like RabbitMQ.
A second approach is to have a "shared" SAN volume that all the application servers will write their logfiles to.
What are the pros/cons of the above approaches? Any caveats we should be wary of? Best practices?