Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a Rails application deployed using Nginx + Unicorn. Last night, when the traffic peaked, Nginx was getting lots of 404 errors. When I dig into logs, those 404 requests on the nginx side did not even get to Rails because I could not find these requests in Rails log. What is the problem here? Rails app was overloaded?

The server is a dual-core 4GB memory virtual host with Ubuntu 11.10 + nginx 1.0.5, the CPU was pretty much packed during the peak traffic. I am using 4 unicorn worker processes at this moment.

Thanks!

nginx.conf

user deployer staff;
worker_processes 4;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
    multi_accept on;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay off;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    underscores_in_headers on;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /var/log/nginx/access.log main;
    error_log /var/log/nginx/error.log warn;

    gzip on;
    gzip_disable "msie6";

    gzip_proxied any;
    gzip_min_length 500;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

##
# Virtual Host Configs
##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

upstream upstream app { server unix:/tmp/app_production.socket fail_timeout=0; }

server {
    client_max_body_size 10m;
    listen       80;
    server_name  xxx.xxx.xxx.xxx;

    #charset koi8-r;
    root /home/deployer/apps/app_production/current/public;
    try_files $uri /system/maintenance.html @app;
    #access_log  /var/log/nginx/log/host.access.log  main;

    location @app {
        proxy_pass http://app;
        proxy_redirect off;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        client_max_body_size       10m;
        client_body_buffer_size    128k;

        proxy_connect_timeout      90;
        proxy_send_timeout         90;
        proxy_read_timeout         90;

        proxy_buffer_size          4k;
        proxy_buffers              4 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # static resource
    location ~ ^/(assets|images|javascripts|stylesheeets|system)/ {
        gzip_static on;
        expires max;
        add_header Cache-Control public;
        add_header Last-Modified "";
        add_header ETag "";

        open_file_cache max=100 inactive=500s;
        open_file_cache_valid 600s;
        open_file_cache_errors on;
        break;
    }
}

And most of the 404 errors in nginx log looks like this

2013/02/21 06:15:40 [error] 1953#0: *7081 connect() to unix:/tmp/app_production.socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET /api/modules HTTP/1.1", upstream: "http://unix:/tmp/app_production.socket:/api/modules", host: "myapp.com"
share|improve this question
    
If the requests don't show up in the rails logs, presumably they never got that far. Look for ways to instrument your web stack to find out where the bottlenecks are (perhaps install a copy of the machine and run some web stress test against it). –  vonbrand Feb 21 '13 at 3:33
    
You should at least post your Nginx config and the logs. Otherwise it will be barely possible to give you an answer. –  mauro.stettler Feb 21 '13 at 4:30
    
@mauro.stettler I have attached nginx confs and sample error log in nginx error log. Thanks a lot! I am not sure if is has anything to do with fail_timeout=0 setting since I only single server for now. –  Ben Feb 21 '13 at 8:21
add comment

1 Answer

You did probably hit the maximum number of open sockets, try increasing worker_rlimit_nofile in nginx and fs.file-max in OS.

share|improve this answer
    
fs.file-max is fine, but I found out that the linux user I use to run the rails and unicorn has a really low ulimit number, soft limit is 1024, hard limit is 4096. This seems to be the problem? And the 404 problems go away once I switch from socket to tcp for unicorn. –  Ben Feb 23 '13 at 12:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.