I'm running a relatively simple VPS (a Media Temple (ve)) with a few PHP-based websites and (eventually) a few Node servers. In order to enable WebSockets support, I'm using HAProxy on port 80, which routes to either nginx or a particular Node process.
I've recently run into a problem, though, where over the course of about 24 hours, my server hits the maximum allowed number of open TCP connections (numtcpsock
in Parallels Power Panel, which is set to 1,000). Running nginx alone does not cause this problem, and I currently have no active Node backend servers. Nginx connects to PHP via a UNIX domain socket (and again, the problem doesn't occur with nginx alone). Any thoughts on what could cause this? My configuration:
global
## 00-base
maxconn 500
nbproc 2
defaults
## 00-base
mode http
frontend all
## 00-ports
bind 0.0.0.0:80
## 10-config
timeout client 86400000
default_backend nginx
backend nginx
## 00-timeouts
timeout http-keep-alive 5000
timeout server 10000
timeout connect 4000
## 10-servers
server main localhost:8000
Thanks in advance!
UPDATE: after a little bit of lsof
ing, I was able to determine that more than 90% of the open TCP sockets are indeed owned by HAProxy, and the overwhelming majority of them are in the CLOSE_WAIT
or FIN_WAIT2
states. Is this an HAProxy bug? It seems like a file descriptor leak of some kind, unless it's a misconfiguration on my part.
UPDATE 2: I've noticed a pattern in the lsof
output. It seems to me that what's happening is nginx is closing an internal connection with HAProxy, but before HAProxy formally closes it, it tries to close the external connection with the client (putting it into FIN_WAIT2
). Because the FIN never comes, the connection between nginx and HAProxy stays in CLOSE_WAIT
forever. Now the only question is: why is this happening?