I have an environment that generates a frankly ludicrous amount of Syslog traffic - this is mostly due to a culture of leaving debug-level logging on in production applications (and changing this tendency is unfortunately off the table and way above my pay grade).
I've already had to move away from UDP due to the sheer volume causing packets to be lost. A smaller centralized syslog environment (many senders, a handful of receivers) is being quickly overrun, with either the individual syslog servers or their backing storage failing multiple times under load.
I'm now in a position to rearchitect this.
What I'm thinking is something like having syslog-ng on my clients (already standard per config management enforcement), sending out the messages formatted as JSON, via an HAProxy cluster, with the messages coming to rest on an Elasticsearch cluster. As a bonus, both HAProxy and Elasticsearch are independently, horizontally scalable with future needs.
I don't immediately foresee any problems with this, but I'd also like a second opinion. Is this a viable plan? Are there better options? Are there any footguns I should be looking for?
Note: Just to be safe, I want to clarify this question is about sending TCP syslog traffic -to- HAProxy to be load balanced, not HAProxy's own logging of its traffic or status