Log transport and aggregation at scale
This is a duplicate question of Log transport and aggregation at scale and http://stackoverflow.com/questions/1737693/whats-the-best-practice-for-centralised-logging, but the answers might differ now :
- The softwares described in 2009 may have changed since (for example Octopussy evolved from version 0.9 to 1.0.5).
- Rsyslog has become the default on most linux distro.
- Requirements have changed (security, software configuration management, ...).
I'd like to ask the following questions :
- How do you centralize, display and archive system logs?
- How would you like to do it now if you had to?
- Most linux distro use rsyslog nowadays, which can provide reliable log transport. But some older unices, network devices and maybe windows box still use old udp rfc-style transport. How did you manage to get reliable transport?
- Storing logs for a few months can represent a huge amount of disk space. How do you store them? rdbms? Compressed and encrypted text files?