I run an online multiplayer game with about 50K simultaneous players at peak time. The game has a custom client-server protocol (deflate-compressed json blobs with some header info) that can be both client initiated request/response pair or server push. Clients connect to an haproxy instance on AWS, which then balances the load between two application servers (written in Java).
Of late, I notice that at peak hours the response to my clients are quite slow. If I ask a client to connect directly to my Java server, it is as fast as I expect it to be. But if the connection goes through the Haproxy instance, a very large latency is introduced. I measure this latency through a custom Nagios plugin I wrote, and at this moment the values stand:
Direct to server A: 135829 microseconds
Direct to server B: 128737 microseconds
Via Haproxy (balanced to one of these servers): 12990736 microseconds
I am stuck at the moment as to what to do next. I'm running haproxy 1.5.5. Config is here: http://pastebin.com/rQ8y6AQj. I have transparent proxying enabled. net.netfilter.nf_conntrack_max is set to 131072. Everything is running on Amazon EC2.