Stack Overflow Careers is served up something like so:
user -> internet -> our fw -> nginx -> haproxy -> web farm
- FreeBSD is the operating system in use
- no firewalling or QoS is in place on this box
- nginx handles our SSL termination
- haproxy handles the load balancing
- nginx / haproxy are pushing about 15 Mbps each way
During normal operation, nginx receives the HTTP request, does its thing, and hands off the request to an haproxy instance that is bound to the loopback address (127.0.0.1) on that same box.
In order to do some troubleshooting the other day, I moved the haproxy instance onto the same interface that nginx was running on. This immediately added 100ms of latency to all requests. This interface is not a true physical interface, but a carp interface.
Can anyone explain to me why this was the case? Contention with the packet queue maybe? Or perhaps loopback is always faster because it is 'soft'? There is something fundamental that I'm missing here, and I'm hoping someone will kindly educate me.