Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have an API that sends messages to RabbitMQ.

I have a high availability RabbitMQ cluster behind HAProxy.

When I load test my API I start seeing lots of this:

Recovering from a network failure... Exception in the reader loop: AMQ::Protocol::EmptyResponseError: Empty response received from the server.

In my unicorn log.

If I connect directly to RabbitMQ as a posed to via haproxy I do not. Where am I going wrong, my haproxy config looks like this:

global
  log 127.0.0.1   local0
  log 127.0.0.1   local1 notice
  #log loghost    local0 info
  maxconn 4096
  #debug
  #quiet
  user haproxy
  group haproxy

defaults
  log     global
  mode    http
  retries 3
  timeout client 50s
  timeout connect 10s
  timeout server 50s
  option dontlognull
  option forwardfor
  option httplog
  option redispatch
  balance  roundrobin

# Set up application listeners here.

listen http_frontend
  bind *:80
  mode http
  default_backend http_backend
  option httpclose
  reqadd X-Forwarded-Proto:\ http

listen https_frontend
  bind *:443 ssl crt /etc/haproxy.pem
  mode http
  default_backend http_backend
  reqadd X-Forwarded-Proto:\ https

listen http_bucky_frontend
  bind *:1880
  mode http
  default_backend http_bucky_backend
  option httpclose
  reqadd X-Forwarded-Proto:\ http

listen https_bucky_frontend
  bind *:1443 ssl crt /etc/haproxy.pem
  mode http
  default_backend http_bucky_backend
  reqadd X-Forwarded-Proto:\ https

listen rabbitmq_frontend
  bind *:5672
  mode tcp
  default_backend rabbitmq_backend
  option tcplog

listen admin
  bind 127.0.0.1:22002
  mode http
  stats uri /


backend http_backend
  mode http
  server 0-http_backend x.x.x.x:9000 maxconn 100 check
  server 1-http_backend x.x.x.x:9000 maxconn 100 check

backend http_bucky_backend
  mode http
  option httpchk GET /status
  http-check expect string up
  server 0-http_bucky_backend x.x.x.x:9000 maxconn 100 check
  server 1-http_bucky_backend x.x.x.x:9000 maxconn 100 check

backend rabbitmq_backend
  balance roundrobin
  mode tcp
  server 0-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check
  server 1-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check

When under load the load balancer is normally 20-30% cpu

share|improve this question
    
I have now determined that the reason for the errors is a timeout as it is happening 50s after the restart HAProxy, which is how long my timeouts are. The question of how to fix remains? –  msaspence Apr 18 at 6:41

1 Answer 1

Remvoe "timeout client 50s" from default section and move to all others except from "rabbitmq_frontend". Client timeout means that it closes TCP session between rabbitmq client and haproxy so reading from this socket gets "".

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.