Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Ubuntu Server 10.04.1 x86

I've got a machine with a FCGI HTTP service behind nginx, that serves a lot of small HTTP requests to a lot of different clients. (About 230 requests per second in the peak hours, average response size with headers is 650 bytes, several millions of different clients per day.)

As a result, I have a lot of sockets, hanging in TIME_WAIT (graph is captured with TCP settings below):

TIME_WAIT

I'd like to reduce the number of sockets.

What can I do besides this?

$ cat /proc/sys/net/ipv4/tcp_fin_timeout
1
$ cat /proc/sys/net/ipv4/tcp_tw_recycle
1
$ cat /proc/sys/net/ipv4/tcp_tw_reuse
1

Update: some details on the actual service layout on the machine:

client -----TCP-socket--> nginx (load balancer reverse proxy) 
       -----TCP-socket--> nginx (worker) 
       --domain-socket--> fcgi-software
                          --single-persistent-TCP-socket--> Redis
                          --single-persistent-TCP-socket--> MySQL (other machine)

I probably should switch load-balancer --> worker connection to domain sockets as well, but the issue about TIME_WAIT sockets would remain — I plan to add a second worker on a separate machine soon. Won't be able to use domain sockets in that case.

share|improve this question
    
It seems that Munin is shamelessly lying. See comments to the Kyle's answer. Looking into that now. –  Alexander Gladysh Dec 13 '10 at 20:52
    
Created a question about Munin: serverfault.com/questions/212200/… –  Alexander Gladysh Dec 13 '10 at 21:25
    
Now it looks like that Munin is not lying, but rather I'm looking at the wrong plot... –  Alexander Gladysh Dec 14 '10 at 1:53
add comment

1 Answer

up vote 15 down vote accepted

One thing you should do to start is to fix the net.ipv4.tcp_fin_timeout=1. That is way to low, you should probably not take that much lower than 30.

Since this is behind nginx. Does that mean nginx is acting as a reverse proxy? If that is the case then your connections are 2x (one to client, one to your web servers). Do you know which end these sockets belong to?

Update:
fin_timeout is how long they stay in FIN-WAIT-2 (From networking/ip-sysctl.txt in the kernel documentation):

tcp_fin_timeout - INTEGER
        Time to hold socket in state FIN-WAIT-2, if it was closed
        by our side. Peer can be broken and never close its side,
        or even died unexpectedly. Default value is 60sec.
        Usual value used in 2.2 was 180 seconds, you may restore
        it, but remember that if your machine is even underloaded WEB server,
        you risk to overflow memory with kilotons of dead sockets,
        FIN-WAIT-2 sockets are less dangerous than FIN-WAIT-1,
        because they eat maximum 1.5K of memory, but they tend
        to live longer. Cf. tcp_max_orphans.

I think you maybe just have to let Linux keep the TIME_WAIT socket number up against what looks like maybe 32k cap on them and this is where Linux recycles them. This 32k is alluded to in this link:

Also, I find the /proc/sys/net/ipv4/tcp_max_tw_buckets confusing. Although the default is set at 180000, I see a TCP disruption when I have 32K TIME_WAIT sockets on my system, regardless of the max tw buckets.

This link also suggests that the TIME_WAIT state is 60 seconds and can not be tuned via proc.

Random fun fact:
You can see the timers on the timewait with netstat for each socket with netstat -on | grep TIME_WAIT | less

Reuse Vs Recycle:
These are kind of interesting, it reads like reuse enable the reuse of time_Wait sockets, and recycle puts it into TURBO mode:

tcp_tw_recycle - BOOLEAN
        Enable fast recycling TIME-WAIT sockets. Default value is 0.
        It should not be changed without advice/request of technical
        experts.

tcp_tw_reuse - BOOLEAN
        Allow to reuse TIME-WAIT sockets for new connections when it is
        safe from protocol viewpoint. Default value is 0.
        It should not be changed without advice/request of technical
        experts.

I wouldn't recommend using net.ipv4.tcp_tw_recycle as it causes problems with NAT clients.

Maybe you might try not having both of those switched on and see what effect it has (Try one at a time and see how they work on their own)? I would use netstat -n | grep TIME_WAIT | wc -l for faster feedback than Munin.

share|improve this answer
    
@Kyle: what value for net.ipv4.tcp_fin_timeout would you recommend? –  Alexander Gladysh Dec 13 '10 at 18:37
    
@Kyle: client --TCP-socket--> nginx (load balancer reverse proxy) --TCP-socket--> nginx (worker) --domain-socket--> fcgi-software –  Alexander Gladysh Dec 13 '10 at 18:39
    
@Kyle: I probably should replace TCP socket between load balancer and worker on the same machine with domain socket as well. –  Alexander Gladysh Dec 13 '10 at 18:40
    
I would say 30 or maybe 20. Try it and see. You have a lot of load so a lot of TIME_WAIT kind of makes sense. –  Kyle Brandt Dec 13 '10 at 18:41
1  
Oh, here is a nice one liner: netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c. So @Alex, if Munin doesn't like up, maybe drill into just how it monitors these stats. Maybe the only issue is that Munin is giving you bad data :-) –  Kyle Brandt Dec 13 '10 at 20:27
show 29 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.