The problem is that from time to time a connection is left in FIN_WAIT2 state.
$ sudo netstat -tpn | grep FIN_WAIT2 tcp6 0 0 10.244.0.1:33132 10.244.0.35:48936 FIN_WAIT2 14125/kube-proxy tcp6 0 0 10.244.0.1:48340 10.244.0.35:56339 FIN_WAIT2 14125/kube-proxy tcp6 0 0 10.244.0.1:52619 10.244.0.35:57859 FIN_WAIT2 14125/kube-proxy tcp6 0 0 10.244.0.1:33132 10.244.0.50:36466 FIN_WAIT2 14125/kube-proxy
These connections stack up over time making the process misbehave. I already reported an issue to Kubernetes bug-tracker but I'd like to understand why such connections are not closed by the Linux kernel.
According to its documentation (search for tcp_fin_timeout) connection in FIN_WAIT2 state should be closed by the kernel after X seconds, where X can be read from /proc. On my machine it's set to 60:
$ cat /proc/sys/net/ipv4/tcp_fin_timeout 60
so if I understand it correctly such connections should be closed by 60 seconds. But this is not the case, they are left in such state for hours.
While I also understand that FIN_WAIT2 connections are pretty unusual (it means the host is waiting for some ACK on from the remote end of the connection which might already be gone) I don't get why these connections are not "closed" by the system.
Is there anything I could do about it?
Note that restarting the related process is a last resort.