8

Stack Overflow Careers is served up something like so:

user -> internet -> our fw -> nginx -> haproxy -> web farm
  • FreeBSD is the operating system in use
  • no firewalling or QoS is in place on this box
  • nginx handles our SSL termination
  • haproxy handles the load balancing
  • nginx / haproxy are pushing about 15 Mbps each way

During normal operation, nginx receives the HTTP request, does its thing, and hands off the request to an haproxy instance that is bound to the loopback address (127.0.0.1) on that same box.

In order to do some troubleshooting the other day, I moved the haproxy instance onto the same interface that nginx was running on. This immediately added 100ms of latency to all requests. This interface is not a true physical interface, but a carp interface.

Can anyone explain to me why this was the case? Contention with the packet queue maybe? Or perhaps loopback is always faster because it is 'soft'? There is something fundamental that I'm missing here, and I'm hoping someone will kindly educate me.

Improve this question
3
  • 1
    A packet sent to an on-box address, whether addressed through lo or an e{th,n} port, never hits the hardware in Linux, regardless. I can't speak authoritatively with regards to BSD, though.
    – BMDan
    Jul 23, 2010 at 15:30
  • Are you sure that you switched it to the same interface? Did the 100ms go away when you switched haproxy back to loopback?
    – tomjedrz
    Jul 23, 2010 at 16:01
  • @tomjedrz - yes. as soon as I switched back, the latency was gone.
    – Michael Gorsuch
    Jul 23, 2010 at 19:56

3 Answers 3

Reset to default
2

A constant 100ms delay looks weird. It looks like packets get buffered and not immediately delivered. Or maybe some of them are dropped and retransmitted. Can you run tcpdump on this interface to show the problem ? I don't know how the IP stack works on FreeBSD, nor how CARP is implemented, but would it be possible for instance that the slave regularly advertises its MAC address with gratuitous ARPs and that the master alternatively sends packets to each side ?

Could you also run tcpdump on the real interface to ensure that nothing gets emitted out ?

Is it possible that the system refrains from caching a CARP device's ARP entry, thus causing an ARP request to be emitted for each packet of a session, that the CARP daemon would have to answer ?

Most of those are some stupid ideas, but it's in order to help you search in the right direction.

Improve this answer
1
  • Thanks for the ideas, Willy. I move the configuration back to the interface off-hours and see what a packet trace turns up.
    – Michael Gorsuch
    Jul 24, 2010 at 21:55
1

Just for clarity, you only changed how it was being accessed, from the 127 address, to the local IP; correct?

If that's the case and it made a difference, something ain't right. Check your routing table with netstat -rn and see what the local IPs are routed to, it should be routed to the lo0 interface (just like 127).

Your netstat -rn output should be vaguely similar to this:

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            1.2.3.1            UGS       131  2655014   nge1
1.2.3.0/23         link#2             U           0       88   nge1
1.2.3.4            link#2             UHS         0    34848    lo0
127.0.0.1          link#5             UH          0    64678    lo0
192.168.0.0/26     link#1             U           2 41703537   nge0
192.168.0.1        link#1             UHS         0    70088    lo0
Improve this answer
5
  • I should've included this in the post: these interfaces are carp interfaces. Completely slipped my mind until I ran netstat. Does that make a difference?
    – Michael Gorsuch
    Jul 23, 2010 at 19:56
  • Yeah, that's it right there. If the address you're using is assigned to a carp interface using that IP will force it through the carp stack before it hits the loopback device; 100ms would be excessive still though. Is the host in question the master of that IP, or are you using load balancing? It could be sending the traffic to the other carp host.
    – Chris S
    Jul 23, 2010 at 20:56
  • The host is the master of that IP.
    – Michael Gorsuch
    Jul 23, 2010 at 22:53
  • I just finished whipping up a similar environment and testing it out. I saw no appreciable difference in response times between the carp interface IP, 127 IP, and a physical IP. I've only got one server here to test, so no carp slaves, but I suspect something's amiss elsewhere in your environment (firewall or traffic shaping?) that's causing the latency. This is a i386-8.1-STABLE.
    – Chris S
    Jul 24, 2010 at 3:49
  • Thanks, Chris. I'll see if I can gather more info when traffic dies down. The current system is not using any firewall package or doing any traffic shaping. I should also note (will update original question) that we are receiving a large amount of traffic due to the job ads that we're displaying on the SO family sites. We're moving about 15 Mbps each way during normal hours.
    – Michael Gorsuch
    Jul 24, 2010 at 21:53
0

I have seen loopback implemented as an interrupt-level software i/f such that traffic never gets outside the box. Could this have been the case when you were running loopback? Disclaimer: Just a general question; I know nothing about FreeBSD.

-- pete

Improve this answer
1
  • This is not the way it works in FreeBSD.
    – Chris S
    Jul 23, 2010 at 15:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .