Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm currently fighting with some crappy piece of (custom) server software which doesn't accept its connections properly (written in Java by a PHP programmer who never before touched sockets let alone threads). My guess is that a thread is dying before the socket is properly accepted in the client thread. I can't be sure and it actually doesn't matter much since the software is currently reimplemented; the old version has to be kept running until the new version goes online, as reliable as possible but without any time and money spent on debugging the old codebase.

The bug manifests itself in the following netstat output; some connections are never transferred from the kernel to use space (that's how I interpret this, better interpretations are welcome):

Proto Recv-Q Send-Q Local Address         Foreign Address         State       PID/Program name
tcp6     228      0 192.0.2.105:1988      46.23.248.10:7925       ESTABLISHED -               
tcp6       0      0 192.0.2.105:1988      221.130.33.37:9826      ESTABLISHED 14741/java      
tcp6       0      0 192.0.2.105:1988      46.23.248.2:5867        ESTABLISHED 14741/java      
tcp6    2677      0 192.0.2.105:1988      221.130.33.37:15688     ESTABLISHED -               
tcp6    3375      0 192.0.2.105:1988      221.130.33.36:3045      ESTABLISHED -               
tcp6   14742      0 192.0.2.105:1988      46.23.248.17:4679       ESTABLISHED -               
tcp6     774      0 192.0.2.105:1988      212.9.19.73:36064       ESTABLISHED -               
tcp6      92      0 192.0.2.105:1988      46.23.248.19:7164       ESTABLISHED -               
tcp6       0      0 192.0.2.105:1988      46.23.248.21:6322       ESTABLISHED 14741/java      
tcp6       0      0 192.0.2.105:1988      221.130.39.216:13937    ESTABLISHED 14741/java      
tcp6    3051      0 192.0.2.105:1988      211.139.145.104:31239   ESTABLISHED -               
tcp6     246      0 192.0.2.105:1988      46.23.248.10:5458       ESTABLISHED -               
tcp6     618      0 192.0.2.105:1988      212.9.19.73:20209       ESTABLISHED -               
tcp6    1041      0 192.0.2.105:1988      46.23.248.18:7424       ESTABLISHED -               
tcp6       0      0 192.0.2.105:1988      46.23.248.10:5065       ESTABLISHED 14741/java      

When this happens and the clients reconnect, they tend to work. But they won't reconnect by itself until they run into a rather long timeout. Since the custom full-duplex protocol in its current incarnation doesn't ack any data sent by the client and the latter doesn't expect any regularly incoming requests from the server, this can be days since the client sends its data happily until the kernel's receive queue runs full. On the server (kernel) side it should be possible to detect stale sockets since the clients send data regularly.

So, assuming my interpretation of this problem is correct, what I wondered was if there is a kernel parameter I can tune which makes the kernel drop/close TCP connections with a RST if they aren't read from by the user space in a timely manner.

Better explanations of what happens here are welcome as well.

share|improve this question

2 Answers 2

You can try tuning TCP keepalive to much shorter values. By default a connection can be idle for two hours before keepalive kicks in.

Exactly what values you should use is really dependent on what your application does and what your users expect or how they interact with it.

share|improve this answer
    
Unfortunately, that doesn't work. The client is a proprietary GSM module (Cinterion BGS2T) which doesn't support sending TCP Keepalives. The server part does theoretically use Keepalives but it looks like the thread dies before that socket option is set (the sockets hang around for hours or days if the server isn't restarted). –  mss Sep 4 '13 at 21:35
    
Did you actually try it? –  Michael Hampton Sep 4 '13 at 21:38
    
Yes, I did. I set the Keepalive timeout down to 15 minutes for testing. –  mss Sep 4 '13 at 21:47
    
You might not have any other option than to soundly berate the developers, then. –  Michael Hampton Sep 4 '13 at 21:48
    
Well, the external developers who built this aren't around anymore for good reason. The code is a clusterfuck. I just wondered if there is anything like a timeout or iptables module I can use to kill stale connections. –  mss Sep 4 '13 at 21:50
up vote 0 down vote accepted

I guess the answer is no.

The problem was solved by replacing the software in question but ideas are still welcome.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.