How to understand random 500 internal errors in Nginx?

Question

Our server is showing a lot of 500 internal error pages for random requests. We know the app is working because our code is unit tested. The server setup is Nginx with Unicorn running a single Rails apps on an Ubuntu 10.4 LTS.

As far as we can tell the requests fail before they are passed to the Unicorn because there are no log entries in Unicorn's log file or our Airbrake (another logging service we are using), but the Nginx's log file does show the 500 status entries. The server is under normal load so it's not running out of RAM or anything.

The Nginx error.log file isn't really giving us anything useful. There are no entries that suggest file issues or resource problems.

How should I go on about finding the issue?

Thanks

Here is our nginx config for the site:

# This is the socket that unicorn listens to
upstream unicorn {
        server unix:/tmp/unicorn.sock;
}

server {
  listen 80;
  client_max_body_size 10m;
  server_name oursite.com;
  root /var/www/current/public;
  access_log  /var/log/nginx/access.log;
  error_page 500 502 503 504 /var/www/shared/500.html;

  if ($http_user_agent ~* (majestic12|easou|Sogou|baidu|ahrefs) ) {
    return 403;
  }

  location / {
#    auth_basic "Restricted";
#    auth_basic_user_file /var/www/shared/.htpasswd;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_max_temp_file_size 0;

    if (-f $request_filename) {
      break;
    }
    if (-f $document_root/system/maintenance.html) {
      rewrite  ^(.*)$  /system/maintenance.html last;
      break;
    }
    if (-f $request_filename/index.html) {
      rewrite (.*) $1/index.html break;
    }
    if (-f $request_filename.html) {
      rewrite (.*) $1.html break;
    }

   if (!-f $request_filename) {
      proxy_pass http://unicorn;
      break;
    }
  }
}

Answer 1

If you have nothing in rails error logs, it probably is HTTP error 503 (unavailable backend) or 504 (backend timeout). Exact error and cause will be in nginx error log. Both of errors are because unicorn cannot serve all HTTP requests.

You can do few things:

• increase worker_processes in unicorn.rb
• increase proxy_read_timeout in nginx.conf
• add second unicorn server and load balance between two by adding it in upstream unicorn