Lots of TIME_WAIT connections with nginx reverse-proxy

Question

I'm using nginx as reverse proxy to redirect some traffic to openfire XMPP server. Here's what I see in netstat:

972 connections to port 80 from outside (includes website and redirected XMPP over http)
of them 231 in TIME_WAIT, seems like normal
272 connection from localhost:<misc port> to localhost:7070 (nginx to xmpp server), all ESTABLISHED
3154 connections from localhost:7070 to localhost:<misc port> (xmpp back to nginx)
of them 2882 in TIME_WAIT (1301 of them with expired timers) seems like it's not normal!

Why there are so many? How to get rid of this connections?

Answer 1

Without more information, it's hard to determine the cause. You can ss to list all connections:

ss --numeric -o state time-wait | less

Count connections by state:

netstat -tan  | awk '{print $6}' | sort | uniq -c

Linux kernel has number of parameters that might influence number of connections:

• sysctl net.ipv4.tcp_keepalive_intv - wait time between isAlive interval probes, default 75, recommended value 15-30
• sysctl net.ipv4.tcp_tw_reuse - allows reusing sockets in TIME_WAIT state for new connections when it is safe from protocol viewpoint. Default value is 0 (disabled).
• sysctl net.ipv4.tcp_fin_timeout - The time that must elapse before TCP/IP can release a closed connection and reuse its resources
• sysctl net.ipv4.tcp_keepalive_probes - The number of probes before timing out

on nginx side each worker process keeps its resource pool, e.g.:

worker_processes 8;
events {
  worker_connections 768;
}
http {
  keepalive_timeout 65;
  server {
     keepalive_requests 1000;
     upstream backend {
        server 127.0.0.1:8080 max_conns=1024;
        # default value: none
        # keepalive 32;
     }
  }
}

see upstream documentation.