HAProxy confusing 503 errors

Question

Almost 1/4 of all requests to a HAProxy instance fail with 503 errors, although the two backend servers are completely healthy and repeated checks on the individual servers return no errors at all.

PHP 5.3 FPM, Nginx, Ubuntu 10.10, HAProxy 1.4.8

Below is my haproxy.cfg:

global
    log 127.0.0.1   local0
    log 127.0.0.1   local1 notice
    #log loghost    local0 info
    maxconn 4096
    #chroot /usr/share/haproxy
    user    haproxy
    group   haproxy
    daemon
    #debug
    #quiet

defaults http
    log         global
    mode        http
    option      httplog
    option      dontlognull
    retries     3
    option      redispatch
    maxconn     2000
    contimeout  5000
    clitimeout  50000
    srvtimeout  50000
    errorfile   400 /etc/haproxy/errors/400.htm
    errorfile   403 /etc/haproxy/errors/403.htm
    errorfile   408 /etc/haproxy/errors/408.htm
    errorfile   500 /etc/haproxy/errors/500.htm
    errorfile   502 /etc/haproxy/errors/502.htm
    errorfile   503 /etc/haproxy/errors/503.htm
    errorfile   504 /etc/haproxy/errors/504.htm

listen www 0.0.0.0:80
    mode    http
    balance roundrobin
    option  redispatch
    cookie  JSESSIONID prefix
    stats   enable
    stats   auth user:pass
    option  httpclose
    option  forwardfor
    option  httpchk HEAD /health HTTP/1.0
    server  web1 0.0.0.0:8080 weight 1 cookie web1 check inter 1000
    server  web2 0.0.0.0:8080 weight 2 cookie web2 check inter 1000

Answer 1

You have a wrong IP address for your servers, both have 0.0.0.0 ! That means that the connection will be forwarded to the same IP haproxy received the connection on, so I can deduce that one of your two servers is running on the same host as haproxy.

Please fix that and check your logs. It will probably not fix your 503 but it will bring you back to a sane configuration which has a chance to work. Then please check your logs if you still see 503 so that we can find why haproxy experienced an issue. Most of the time a 503 is caused by the impossibility to connect to a server. It's really possible that you're sending them more connection than they can accept. In this case, use the "maxconn" setting on your "server" lines, this will enable traffic regulation to prevent overflowing.