Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm running a relatively simple VPS (a Media Temple (ve)) with a few PHP-based websites and (eventually) a few Node servers. In order to enable WebSockets support, I'm using HAProxy on port 80, which routes to either nginx or a particular Node process.

I've recently run into a problem, though, where over the course of about 24 hours, my server hits the maximum allowed number of open TCP connections (numtcpsock in Parallels Power Panel, which is set to 1,000). Running nginx alone does not cause this problem, and I currently have no active Node backend servers. Nginx connects to PHP via a UNIX domain socket (and again, the problem doesn't occur with nginx alone). Any thoughts on what could cause this? My configuration:

global
    ## 00-base
    maxconn     500
    nbproc      2
defaults
    ## 00-base
    mode        http
frontend all
    ## 00-ports
    bind 0.0.0.0:80
    ## 10-config
    timeout client 86400000
    default_backend nginx
backend nginx
    ## 00-timeouts
    timeout http-keep-alive 5000
    timeout server 10000
    timeout connect 4000
    ## 10-servers
    server main localhost:8000

Thanks in advance!

UPDATE: after a little bit of lsofing, I was able to determine that more than 90% of the open TCP sockets are indeed owned by HAProxy, and the overwhelming majority of them are in the CLOSE_WAIT or FIN_WAIT2 states. Is this an HAProxy bug? It seems like a file descriptor leak of some kind, unless it's a misconfiguration on my part.

UPDATE 2: I've noticed a pattern in the lsof output. It seems to me that what's happening is nginx is closing an internal connection with HAProxy, but before HAProxy formally closes it, it tries to close the external connection with the client (putting it into FIN_WAIT2). Because the FIN never comes, the connection between nginx and HAProxy stays in CLOSE_WAIT forever. Now the only question is: why is this happening?

share|improve this question

2 Answers 2

up vote 1 down vote accepted

The issue is caused by your extremely large timeout. With a 24-hours timeout and a limit to 1000 concurrent connections, you can clearly expect to fill this with clients disconnecting the dirty way. Please use a more reasonable timeout, from minutes to hours at most, it really makes no sense to use 1 day timeouts on the internet. As DukeLion said, the system is waiting for haproxy to close the connection, because haproxy did not receive the close from the client.

Haproxy being working in tunnel mode for TCP and WebSocket, it follows the usual 4-way close :

- receive a close on side A
- forward the close on side B
- receive the close on side B
- forward the close on side A

In your case, I suppose that side A was the server and side B the client. So nginx closed after some time, socket went to CLOSE_WAIT, haproxy forwarded the close to the client, this socket went to FIN_WAIT1, the client ACKed, passing the socket to FIN_WAIT2 and then nothing happens because the client has disappeared, which is something very common on the net. And your timeout means you want this to remain this way for 24 hours.

After 24 hours, your sessions will start timing out on the client side so haproxy will kill them and forward the close to the nginx side, getting rid of it too. But clearly you don't want this to happen, WebSocket was designed so that idle connection could be reopened transparently, so there is no reason to keep an idle connection open for 24 hours. No firewall will keep it along the way !

share|improve this answer
    
That'll do it! I figured the client timeout was too high, especially for ordinary websites (I copied it from an example online somewhere), but thought it might be necessary for WebSockets. Since connections get transparently maintained, though, that's clearly not the case! Thanks! –  Matt Patenaude May 21 '12 at 21:45

many sockets in CLOSE_WAIT is extremely bad thing for your server. This states happens when kernel waits for userspace software do accept socket closing. If many sockets stays in this state for a long time - it means, that software using it is unresponsive. Usually socket in this state consumes relatively high amount of kernel cpu time.

I suppose that CLOSE_WAITs in your case is secondary to FIN_WAIT2 - HAproxy is waiting for client connection to close, and closing connection with NGINX just after that.

FIN_WAIT2 is state of waiting other side to confirm socket closing. It's not that bad to have many of them, but it may indicate network congestion or heavy losses.

You can try nolinger option with haproxy to close connection faster. But be carefull, this will break tcp mechanism of guaranteed delivery.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.