Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a web service running behind an instance of nginx running as a reverse proxy. Nginx is configured to load balance between 10 app server processes running on two separate hosts.

The problem I'm seeing is that at around 150 concurrent connections nginx starts rejecting all new incoming connections. At 140 connections everything is fast and stable, adding just a few more the server will start rejecting all incoming connections not just those above 150. Once all connections have been broken, it will start accepting again.

This does not seem to change at all as I modify worker_processes, worker_connections or multi_accept settings. When the rejections start, there is very little CPU load (>10%) and plenty of network bandwidth available. There are no messages in the error logs.

What am I doing wrong here?

Here's the config:

worker_processes 8;
worker_rlimit_nofile 65536;

events {
    worker_connections  8192;
    multi_accept on;
    use epoll;
}

http {
    include       /etc/nginx/mime.types;
    access_log  /var/log/nginx/access.log;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

In the conf.d directory there are just mappings from host names to backend servers. Like this:

upstream api {
    server 10.0.0.1:8000;
    server 10.0.0.1:8001;
    server 10.0.0.2:8000;
    server 10.0.0.2:8001;
}

server {
    listen 80;
    server_name api.example.com;
    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_redirect off;
        proxy_pass http://api;
    }
}
share|improve this question
    
Whats in the included configs? :) –  3molo Jun 15 '11 at 15:46
    
How does the rejection occur? Error page, connection timeout, connection reset? –  Shane Madden Jun 15 '11 at 15:58
    
The connections are closed by the server. –  joegester Jun 15 '11 at 16:19
    
All that's in the included configs is the hostname to backend mappings. I didn't want to post the urls and ips and stuff. I'll add a redacted version to the question. –  joegester Jun 15 '11 at 16:21
    
It matters a lot if you proxy, fastcgi etc etc –  3molo Jun 15 '11 at 16:27

5 Answers 5

What does vmstat 1 say during the problem occurence ? Is the server loaded in any way during the fault ?

Also check what is the real number of open connections on the server (lsof -i | grep nginx or netstat -atnp | grep nginx ). It could be the problem of connections to app servers ( maybe some kind of connection overflow to backend)

It could be ulimit problem (you would see number of open files with lsof | grep nginx).

Last thing I would try is stracing the nginx process while the problem occurs.

share|improve this answer

I had the very same issue, and found a similar question here.

The proposed solution worked for me, after restarting nginx.

share|improve this answer
    
Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. –  Scott Pack Oct 19 '12 at 22:47

Well, I have two major ideas. My first bet would be to check the system limits. It could be that nginx is running out of file descriptors, or it could be the kernel is forbidding nginx from taking more than a certain number of connections. If something's happening at the kernel level, nginx might not ever know about it. Have you checked all the usual daemon logs?

But, honestly, I suspect that it's more likely related to your fastcgi comparison. So, for a really easy test, use ab (apachebench) to hit a static file on the nginx proxy, and hit it with a couple hundred simultaneous connections for a few thousand times. My guess is that it will do it with no trouble at all. That means you're probably running into queueing issues on the fastcgi side, and nginx is just dropping connections on the floor instead of waiting for fastcgi to return.

share|improve this answer

What about open file limit? 150 still seems to little to hit even default limit of 1024, but check output of ulimit -n. You might need to increase limit in init.d script (using ulimit command) or in /etc/security/limits.conf

share|improve this answer

Is this on a micro instance on EC2 by any chance?

I ran into this problem last week and found this article: http://gregsramblings.com/2011/02/07/amazon-ec2-micro-instance-cpu-steal/

The situation was resolved by going to a small instance type.

Maybe this will help!

share|improve this answer
    
Nope. This is three physical servers. –  joegester Jun 15 '11 at 16:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.