I have cluster of several machines connected on 10GBE network (NICs are Intel 82599EB 10GBE SFI/SFP+) running under Debian 6.0 and faced with problem of hung up TCP connections in CLOSE_WAIT state. I know that in theory connection in CLOSE_WAIT state should be explicitly closed by application, but in my case at least two different applications generate these hung up connections and I think that problem is in something else.
At first this problem was reproduced by Cassandra running as daemon under 'jsvc' process. One node of Cassandra ("server") didn't close connection that was closed one the side of another node that initiated this connection ("client"). After that I ran 'netperf' TCP_CRR test and got error message:
netperf -H 184.108.40.206 -t TCP_CRR -l -5 -D TCP Connect/Request/Response TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 220.127.116.11 (18.104.22.168) port 0 AF_INET : demo send_tcp_conn_rr: data recv error: Connection reset by peer
With TCP connaction hanging in CLOSE_WAIT state on the 22.214.171.124 machine with strange 1 byte in Recv-Q.
tcp 1 0 126.96.36.199:12865 188.8.131.52:42863 CLOSE_WAIT
I've updated 'ixgbe' driver to the latest 3.9-NAPI, but this problem still persists and now I wonder what else can cause the problem?