Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

There is Debian Squeeze servers:

1) Load Balancer - Haproxy 1.4.18, 16 cpu cores, 5-10 GB RAM (it's cloud service)

2) 4 Apache backends (10 GB RAM, 16 cpu cores)

When we start haproxy, site works well for about 2 minutes, then for about 30 seconds it hangs. (At this moment direct access to backends works well, thereis no slow database queries or any backend problems) After that situation repeats. Interesting observation: we tried to use nginx as balancer and we got the SAME behaviour.

When haproxy (nginx) hang, there is no any cpu or ram highload.

What can be reason of this behaviour?

Thanks!

share|improve this question

4 Answers 4

I had symptoms like that and it turned out that I was hitting the connection limit (something like 2000 connections, which is small enough to be used up by ~300 concurrent users browsing the site, no idea why the limit is so small). I've found that the following in /etc/haproxy/haproxy.cfg made my site a lot smoother:

global
    maxconn 8192

defaults
    [...]
    option  httpclose
    maxconn 8192
share|improve this answer
    
We have very big connection limit at haproxy.cfg (more than 10000). I think the problem is in some sort of system tcp (?) limits, because problem repeats with nginx –  Vitaly Baev Oct 13 '11 at 10:17
up vote 1 down vote accepted

Problem solved. This issue appears because there are following strings by haproxy in syslog:

paroxy: nf_conntrack: table full, dropping packet
paroxy: nf_conntrack: table full, dropping packet
paroxy: nf_conntrack: table full, dropping packet
paroxy: nf_conntrack: table full, dropping packet

We changed sysctl settings from

sysctl -a | grep conntrack_max
net.ipv4.netfilter.ip_conntrack_max = 65536
net.netfilter.nf_conntrack_max = 65536
net.nf_conntrack_max = 65536

to the bigger value.

share|improve this answer
    
It's usually recommended to not use the conntrack module with haproxy, due to this very reason. You will lose the stateful properties of iptables, but you just have to adjust your rules accordingly. –  JimB Nov 30 '11 at 17:52

How many requests do you have between nginx and Apache's servers? Isn't there any waiting connections?

I had few problems with that, when i had many requests between them. Try to disable keep-alive connections on Apache's and nginx, if you can.

share|improve this answer
    
Thanks for answer! Thereis about 100-200 requests between load balancer and apache. In Apache (and nginx) config keep-alive disabled. –  Vitaly Baev Oct 12 '11 at 23:22

what does your haproxy.cfg look like ? any firewalls in place/rate limiting ?

how was haproxy installed ? configure options ?

tried using 1.5 dev7 haproxy version with make TARGET=linux26, don't seem to have any problems on my site - benchmarked at http://centminmod.com/siegebenchmark_nginx_test3.html

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.