Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a decently high trafficked FreeBSD Nginx server, and I'm starting to get a large number of listen queue overflows:

[root@svr ~]# netstat -sp tcp | fgrep listen
80361931 listen queue overflows
[root@svr ~]# netstat -Lan | grep "*.80"
tcp4  192/0/128        *.80
[root@svr ~]# sysctl kern.ipc.somaxconn
kern.ipc.somaxconn: 12288
[root@svr ~]#

However I can't seem to increase the max listen queue length past 128. I've increased kern.ipc.somaxconn, but it's not changing the max. Am I missing something?

Thanks!

share|improve this question

2 Answers 2

The kern.ipc.somaxconn might not do what you think it does. It is the limit to outstanding and unhandled connections. (e.g.. it is not a connection limit, but the pending handle connection limit).

To use a non computer analogy: It is the maximum number of ringing phones (before they are picked up and answered), not the number of maximum simultaneous phone calls.

If you have backlog that is that big then your need to let your application pick up the phone more often (e.g. give it more resources, more CPU, better framework, whatever).

Note that the FreeBSD handbook section on kernel limits states this: (Emphasis mine).

The kern.ipc.somaxconn sysctl variable limits the size of the listen queue for accepting new TCP connections. The default value of 128 is typically too low for robust handling of new connections in a heavily loaded web server environment. For such environments, it is recommended to increase this value to 1024 or higher. The service daemon may itself limit the listen queue size

I have no experience with Nginx, but also check its configuration files for above mentioned limit from the application side.

share|improve this answer
    
Thanks for the answer - I'm aware it's a queue, and during spikes it's too much for Nginx to handle, however there are sufficient resources to process it (even after a second or two delay, which is fine). I don't believe this is an Nginx problem because netstat is a system command; the OS is showing the queue overflows (OS level, not app level). My issue is as simple as how to increase the 128 max listen queue size... I believe that'll solve my problem, I just can't seem to get FreeBSD to increase it. Thanks! –  Harry Oct 29 '12 at 16:25
    
The OS is the part where the connection needs to be processed first (before it can hand it over to the application). So yes, if the apps can not keep up it will get dropped at OS level. If you increase OS buffers so it can buffer spikes then that should work. If Nginx is consistently slower then it is not going to work. But for brief spikes, aye, I think it should work. –  Hennes Oct 29 '12 at 16:30
    
Exactly - the purpose of this question is to try and figure out how to increase the OS buffer, so it can absorb small spikes. I haven't figured out how to increase the 128 tcp max from netstat -Lan. –  Harry Oct 29 '12 at 16:47
    
When a TCP Server opens a socket to accept incoming commands it does so with listen() which takes two arguments: 1. the socket on which to listen 2. The number of backlogged connections allowed. The error message you're getting indicates either: 1. the system level limit has been reached, specified by kern.ipc.somaxconn 2. the application level limit has been reached, specified by the application in the listen() call. I'm not familiar with nginx and too lazy to go diving into it's poor documentation or uncommented source code. –  Chris S Oct 29 '12 at 17:00
1  
Yes, changeing kern.ipc.somaxconn will increase the system limit. It will not affect any applications that are currently running. Applications would have to close listening sockets and re-open them at a minimum to change their limit. Nginx documentation say their limit is "-1" by default, which tells the OS to use the system limit. I haven't dug through any source code or bug reports to verify, but it sounds reasonable. –  Chris S Oct 29 '12 at 17:11

May be listen queue limited to 128 in nginx config

Look nginx config for settings like:

listen 80 backlog=128;

And either remove backlog (default is -1 = use system limit) or change to bigger value (8192 should be enough even for loaded server). If even with increased listen queue your still see listen queue overflow it may indicate, that nginx is blocked for a long time (because of slow/overloaded HDD, or by bad written 3rd-party module).

share|improve this answer
1  
Also, did you restart nginx after changing kern.ipc.somaxconn? –  SaveTheRbtz Oct 30 '12 at 7:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.