Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I don't know how troubleshoot this. I get "503 Service Unavailable" http error for all "nginx upstreams" proxy passing calls to haproxy fast_thin and slow_thin ( server 127.0.0.1:3100 and server 127.0.0.1:3200 ), which loadbalance on 6 Thin servers ( 127.0.0.1:3000 .. 3005 ). Static files like /blog are currently fine. The falldown is: nginx on port 80 - haproxy on 3100 and 3200 - thin on 3000 .. 3005 and then Rails. Here it is /etc/nginx/nginx.conf :

user  nginx;
worker_processes  2;

pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    sendfile        on;
    tcp_nopush     on;

    keepalive_timeout  65;
    tcp_nodelay        on;

    include /etc/nginx/conf.d/*.conf;
}

then /etc/nginx/conf.d/default.conf

upstream fast_thin {
    server 127.0.0.1:3100;
}
upstream slow_thin {
    server 127.0.0.1:3200;
}
server {
            listen   80;
            server_name  www.gitwatcher.com;
            rewrite ^/(.*) http://gitwatcher.com/$1 permanent;
}
server {
    listen   80;
    server_name gitwatcher.com;

    access_log /var/www/gitwatcher/log/access.log;
    error_log  /var/www/gitwatcher/log/error.log;

    root       /var/www/gitwatcher/public;
    # index      index.html;

            location /about {
              proxy_pass http://fast_thin;
              break;
        }

            location /trends {
              proxy_pass http://slow_thin;
              break;
        }
            location /categories {
                          proxy_pass http://slow_thin;
                          break;
            }
            location /signout {
                          proxy_pass http://slow_thin;
                          break;
            }
            location /auth/github {
                          proxy_pass http://slow_thin;
                          break;
            }             


        location / {
                          proxy_set_header  X-Real-IP  $remote_addr;
                          proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
                          proxy_set_header Host $http_host;
                          proxy_redirect off;

                          if (-f $request_filename/index.html) {
                                           rewrite (.*) $1/index.html break;
                          }

                          if (-f $request_filename.html) {
                                           rewrite (.*) $1.html break;
                          }

                          if (!-f $request_filename) {
                                           proxy_pass http://slow_thin;
                                           break;
                          }
            }
}

then haproxy config file /etc/haproxy/haproxy.cfg :

global

  log 127.0.0.1   local0
  log 127.0.0.1   local1 notice
  #log loghost    local0 info
  maxconn 4096
  #chroot /usr/share/haproxy
  user haproxy
  group haproxy
  daemon
  #debug
  #quiet
  nbproc        1         # number of processing cores

defaults

        log     global
        retries 3
        maxconn 2000
        contimeout      5000

  mode              http
  clitimeout        60000       # maximum inactivity time on the client side
  srvtimeout        30000       # maximum inactivity time on the server side
  timeout connect   4000        # maximum time to wait for a connection attempt to a server to succeed

        option  httplog
        option  dontlognull
        option redispatch

  option            httpclose     # disable keepalive (HAProxy does not yet support the HTTP keep-alive mode)
  option            abortonclose  # enable early dropping of aborted requests from pending queue
  option            httpchk       # enable HTTP protocol to check on servers health
  option            forwardfor    # enable insert of X-Forwarded-For headers

  balance roundrobin            # each server is used in turns, according to assigned weight

  stats enable                  # enable web-stats at /haproxy?stats
  stats auth        haproxy:pr0xystats  # force HTTP Auth to view stats
  stats refresh     5s      # refresh rate of stats page

listen rails_proxy 127.0.0.1:3100

  # - equal weights on all servers
  # - maxconn will queue requests at HAProxy if limit is reached
  # - minconn dynamically scales the connection concurrency (bound my maxconn) depending on size of HAProxy queue
  # - check health every 20000 microseconds

  server web1 127.0.0.1:3000 weight 1 minconn 3 maxconn 6 check inter 20000
  server web1 127.0.0.1:3001 weight 1 minconn 3 maxconn 6 check inter 20000
  server web1 127.0.0.1:3002 weight 1 minconn 3 maxconn 6 check inter 20000

listen slow_proxy 127.0.0.1:3200

  # cluster for slow requests, lower the queues, check less frequently

  server slow1 127.0.0.1:3003 weight 1 minconn 1 maxconn 3 check inter 40000
  server slow2 127.0.0.1:3004 weight 1 minconn 1 maxconn 3 check inter 40000
  server slow3 127.0.0.1:3005 weight 1 minconn 1 maxconn 3 check inter 40000

and the Thin config file /etc/thin/gitwatcher.yml :

---
chdir: /var/www/gitwatcher
environment: production
address: 0.0.0.0
port: 3000
timeout: 300
log: log/thin.log
pid: tmp/pids/thin.pid
max_conns: 1024
max_persistent_conns: 100
require: []
wait: 30
servers: 3
daemonize: true

if I look into open listen ports, I got the following :

root@fullness:/var/www/gitwatcher# lsof | grep TCP | egrep "nginx|haproxy|thin"
nginx      834       root    8u     IPv4                921        0t0        TCP *:http (LISTEN)
nginx      835      nginx    8u     IPv4                921        0t0        TCP *:http (LISTEN)
nginx      837      nginx    8u     IPv4                921        0t0        TCP *:http (LISTEN)
haproxy   1908    haproxy    4u     IPv4              11699        0t0        TCP localhost:3100 (LISTEN)
haproxy   1908    haproxy    6u     IPv4              11701        0t0        TCP localhost:3200 (LISTEN)
root@fullness:/var/www/gitwatcher# 

iptables -L get me the following :

Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:22222
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
ACCEPT     all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            

Any help ?

UPDATE: trying troubleshooting the way suggested doesn't help because I simply get a "couldn't connect" error, no more :

root@fullness:/var/www/gitwatcher/log# ps -ef | grep thin
root      3740     1  4 15:20 ?        00:00:19 thin server (0.0.0.0:3000)                                                                                                                                                                                                                                                      
root      3809     1  5 15:20 ?        00:00:22 thin server (0.0.0.0:3001)                                                                                                                                                                                                                                                      
root      3834     1  6 15:20 ?        00:00:26 thin server (0.0.0.0:3002)                                                                                                                                                                                                                                                      
root      4166  2274  2 15:27 pts/1    00:00:00 grep --color=auto thin
root@fullness:/var/www/gitwatcher/log# curl http://localhost:3000/trends
curl: (7) couldn't connect to host
root@fullness:/var/www/gitwatcher/log# curl http://localhost:3001/trends
curl: (7) couldn't connect to host
root@fullness:/var/www/gitwatcher/log# curl http://localhost:3002/trends
curl: (7) couldn't connect to host
root@fullness:/var/www/gitwatcher/log# 

UPDATE :

root@fullness:/var/www/gitwatcher# netstat -a | egrep "3000|3001|3002"
tcp        0      0 *:3000                  *:*                     LISTEN     
tcp        0      0 *:3001                  *:*                     LISTEN     
tcp        0      0 *:3002                  *:*                     LISTEN     
root@fullness:/var/www/gitwatcher# 
share|improve this question
    
Your list of listening ports doesn't show 3000-3002 despite the fact that the processes are configured for these ports. Are any errors logged / thrown to the console when you start them? –  Dave Forgac Oct 25 '12 at 16:18
    
yeah, lsof doesn't show thin ports ( don't know way ) but netstat it does, see last UPDATE Is it something wrong with that ? –  Luca G. Soave Oct 25 '12 at 20:57
    
no error on console –  Luca G. Soave Oct 25 '12 at 21:00
add comment

1 Answer

up vote 1 down vote accepted

If you're trying to diagnose 503s on the upstream do one of the following:

On the local box, use curl to request a valid URL on one of the upstreams directly and see the results there:

curl http://localhost:3000/example

If you can't do this, update the configs temporarily so the upstreams listen on an external IP and visit the IP:port in a browser.

Then take a look at the logs for your request. This should allow you to determine where the server error exists.

share|improve this answer
    
thanks Dave, please see UPDATE –  Luca G. Soave Oct 25 '12 at 15:32
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.