0

There is Debian Squeeze servers:

1) Load Balancer - Haproxy 1.4.18, 16 cpu cores, 5-10 GB RAM (it's cloud service)

2) 4 Apache backends (10 GB RAM, 16 cpu cores)

When we start haproxy, site works well for about 2 minutes, then for about 30 seconds it hangs. (At this moment direct access to backends works well, thereis no slow database queries or any backend problems) After that situation repeats. Interesting observation: we tried to use nginx as balancer and we got the SAME behaviour.

When haproxy (nginx) hang, there is no any cpu or ram highload.

What can be reason of this behaviour?

Thanks!

4 Answers 4

1

I had symptoms like that and it turned out that I was hitting the connection limit (something like 2000 connections, which is small enough to be used up by ~300 concurrent users browsing the site, no idea why the limit is so small). I've found that the following in /etc/haproxy/haproxy.cfg made my site a lot smoother:

global
    maxconn 8192

defaults
    [...]
    option  httpclose
    maxconn 8192
1
  • We have very big connection limit at haproxy.cfg (more than 10000). I think the problem is in some sort of system tcp (?) limits, because problem repeats with nginx Oct 13, 2011 at 10:17
1

Problem solved. This issue appears because there are following strings by haproxy in syslog:

paroxy: nf_conntrack: table full, dropping packet
paroxy: nf_conntrack: table full, dropping packet
paroxy: nf_conntrack: table full, dropping packet
paroxy: nf_conntrack: table full, dropping packet

We changed sysctl settings from

sysctl -a | grep conntrack_max
net.ipv4.netfilter.ip_conntrack_max = 65536
net.netfilter.nf_conntrack_max = 65536
net.nf_conntrack_max = 65536

to the bigger value.

1
  • It's usually recommended to not use the conntrack module with haproxy, due to this very reason. You will lose the stateful properties of iptables, but you just have to adjust your rules accordingly.
    – JimB
    Nov 30, 2011 at 17:52
0

How many requests do you have between nginx and Apache's servers? Isn't there any waiting connections?

I had few problems with that, when i had many requests between them. Try to disable keep-alive connections on Apache's and nginx, if you can.

1
  • Thanks for answer! Thereis about 100-200 requests between load balancer and apache. In Apache (and nginx) config keep-alive disabled. Oct 12, 2011 at 23:22
0

what does your haproxy.cfg look like ? any firewalls in place/rate limiting ?

how was haproxy installed ? configure options ?

tried using 1.5 dev7 haproxy version with make TARGET=linux26, don't seem to have any problems on my site - benchmarked at http://centminmod.com/siegebenchmark_nginx_test3.html

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .