Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have an API that sends messages to RabbitMQ.

I have a high availability RabbitMQ cluster behind HAProxy.

When I load test my API I start seeing lots of this:

Recovering from a network failure... Exception in the reader loop: AMQ::Protocol::EmptyResponseError: Empty response received from the server.

In my unicorn log.

If I connect directly to RabbitMQ as a posed to via haproxy I do not. Where am I going wrong, my haproxy config looks like this:

global
  log 127.0.0.1   local0
  log 127.0.0.1   local1 notice
  #log loghost    local0 info
  maxconn 4096
  #debug
  #quiet
  user haproxy
  group haproxy

defaults
  log     global
  mode    http
  retries 3
  timeout client 50s
  timeout connect 10s
  timeout server 50s
  option dontlognull
  option forwardfor
  option httplog
  option redispatch
  balance  roundrobin

# Set up application listeners here.

listen http_frontend
  bind *:80
  mode http
  default_backend http_backend
  option httpclose
  reqadd X-Forwarded-Proto:\ http

listen https_frontend
  bind *:443 ssl crt /etc/haproxy.pem
  mode http
  default_backend http_backend
  reqadd X-Forwarded-Proto:\ https

listen http_bucky_frontend
  bind *:1880
  mode http
  default_backend http_bucky_backend
  option httpclose
  reqadd X-Forwarded-Proto:\ http

listen https_bucky_frontend
  bind *:1443 ssl crt /etc/haproxy.pem
  mode http
  default_backend http_bucky_backend
  reqadd X-Forwarded-Proto:\ https

listen rabbitmq_frontend
  bind *:5672
  mode tcp
  default_backend rabbitmq_backend
  option tcplog

listen admin
  bind 127.0.0.1:22002
  mode http
  stats uri /


backend http_backend
  mode http
  server 0-http_backend x.x.x.x:9000 maxconn 100 check
  server 1-http_backend x.x.x.x:9000 maxconn 100 check

backend http_bucky_backend
  mode http
  option httpchk GET /status
  http-check expect string up
  server 0-http_bucky_backend x.x.x.x:9000 maxconn 100 check
  server 1-http_bucky_backend x.x.x.x:9000 maxconn 100 check

backend rabbitmq_backend
  balance roundrobin
  mode tcp
  server 0-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check
  server 1-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check

When under load the load balancer is normally 20-30% cpu

share|improve this question
    
I have now determined that the reason for the errors is a timeout as it is happening 50s after the restart HAProxy, which is how long my timeouts are. The question of how to fix remains? –  msaspence Apr 18 at 6:41

2 Answers 2

Remvoe "timeout client 50s" from default section and move to all others except from "rabbitmq_frontend". Client timeout means that it closes TCP session between rabbitmq client and haproxy so reading from this socket gets "".

share|improve this answer

Nerijus is right, this issue is caused by the HAProxy having a client timeout, which means, if the connection is considered idle for more than X ms then the connection is dropped.

TCP can send keep-alive packets to ensure an idle connection should stay open.

You can check your TCP parameter for keep-alive packets using the following command:

$ cat /proc/sys/net/ipv4/tcp_keepalive_time

By default, this configuration is equal to 7200 seconds, which means TCP will begin to send keep-alive packets only after a connection is idle for more than 2 hours.

So, just update your HAProxy client timeout value to something > 2 hours, e.g:

timeout client 3h

And add the clitcpka option to your backend:

backend rabbitmq_backend balance roundrobin mode tcp option clitcpka server 0-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check server 1-rabbitmq_backend x.x.x.x:5672 maxconn 4000 check

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.