Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am trying to configure HAProxy to do load balancing for a custom webserver I created. Right now I am noticing an increasing delay with HAProxy as the size of the return message increases. For example, I ran four different tests, here are the results:

Response 15kb through HAProxy:
Avg. response time: .34 secs
Transacation rate: 763 trans/sec
Throughput: 11.08 MB/sec

Response 2kb through HAProxy:
Avg. response time: .08 secs
Transaction rate: 1171 trans / sec
Throughput: 2.51 MB/sec

Response 15kb directly to server:
Avg. response time: .11 sec
Transaction rate: 1046 trans/sec
throughput: 15.20 MB/sec

Response 2kb directly to server:
Avg. Response time: .05 secs
Transaction rate: 1158 trans/sec
Throughput: 2.48 MB/sec

All transactions are HTTP requests. As you can see, there seems to be a much bigger difference between response times for when the response is bigger, than when it is smaller. I understand there will be a slight delay when using HAProxy. Not sure if it matters, but the test itself was run using siege. And during the test there was only one server behind the HAProxy(the same that was used in the direct to server tests). Here is my haproxy.config file:

global  
       log 127.0.0.1   local0
       log 127.0.0.1   local1 notice
       maxconn 10000
       user haproxy
       group haproxy
       daemon
       #debug

defaults
       log     global
       mode    http
       option  httplog
       option  dontlognull
       retries 3
       option redispatch
       option httpclose
       maxconn 10000
       contimeout      10000
       clitimeout      50000
       srvtimeout      50000
       balance roundrobin
       stats enable
       stats uri /stats

listen  lb1  10.1.10.26:80
        maxconn 10000
        server  app1 10.1.10.200:8080 maxconn 5000

I couldn't find much in terms of options in this file that would help my problem. I have heard suggestions that I may have to adjust a few of my sysctl settings. I could not find a lot of information on this however, most documentation is for Linux 2.4 and 2.6 on the sysctl stuff, I am running 3.2(Ubuntu server 12.04), which seems to auto tuning, so I have no clue what I should or shouldn't be changing. Most settings changes I tried had no effect or a negative effect on performance.

Just a notice, this is a very preliminary test, and my hope is that at deployment time, my HAProxy will be able to balance 10k-20k requests/sec to many servers, so if anyone could provide information to help me reach that goal, it would be much appreciated.

Thank you very much for any information you can provide. And if you need anymore information from me please let me know, I will get you anything I can.

[Edit] As requested haproxy -vv

HA-Proxy version 1.4.18 2011/09/16
Copyright 2000-2011 Willy Tarreau <w@1wt.eu>

Build options :
  TARGET  = linux26
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing
  OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes

Available polling systems :
   sepoll : pref=400,  test result OK
   epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
   select : pref=150,  test result OK
Total: 4 (4 usable), will use sepoll.
share|improve this question
    
please provide the output of haproxy -vv –  longneck Sep 26 '12 at 21:24
    
Updated original post with information. Confused why my maxconn limit is 2000 when in my config file it is 10000. –  Eumcoz Sep 26 '12 at 21:41
    
that's the default setting, i.e. the value used when you don't specify one in your configuration file. –  longneck Sep 26 '12 at 22:09
    
I see, do you see anything wrong with the output that would cause the behavior shown? –  Eumcoz Sep 26 '12 at 22:10
    
i don't, but smarter eyesballs will be along shortly. –  longneck Sep 26 '12 at 22:11

1 Answer 1

up vote 2 down vote accepted

I'm thinking about several points :

1) are you running in a virtualized environment ?

2) do you have nf_conntrack loaded on the machine ?

3) did you at any point saturate the CPU on any of the machines involved ?

4) please use "option http-server-close" instead of "option httpclose", as the latter lets both sides close by themselves, resulting in longer connections.

5) what do you see in haproxy's logs ? The time will be split in multiple fields that let you analyse where it is spent.

6) if you see much smaller times in haproxy's logs than what you have on your test machine, it means the delay is caused SYN packets waiting in the system's backlog (or worse, retransmitted), which could be cause by a lack of system tuning.

7) (less important), when reporting an issue with an old version, you should first update it to the latest fixes (1.4.22) to see if the issue is still there. I don't think what you observe matches any known issue but still that's the general idea.

share|improve this answer
    
I did fix it but have a couple questions: 1) Yes it is. Can HAProxy not be run on a virtualized environment well? In the future I am looking to move this to an Amazon EC2 instance, would something like this be a problem? 2) I do not, is it something I should install that helps performance? 5)I cannot seem to find log files, is there something I need to do to enable this on Ubuntu 12.04. 7)Fixed, thought I was running 1.4.22. Moving off a virtualized environment and upgrading to 1.4.22 fixed it(did not try them separately). Thank you for the help, it is very appreciated. –  Eumcoz Oct 2 '12 at 16:15
1  
1) virtualized environments are notoriously known for adding long random delays, sometimes multiple milliseconds. Run a ping between 2 EC2 instances and you'll see. 2) no, do not use nf_conntrack, it could even reduce performance, I was just checking. 5) logs: you need to enable them via your syslog daemon, it's also explained in the doc if you want. –  Willy Tarreau Oct 14 '12 at 9:28
    
Okay, that helps a lot for planning as we go forward. Thank you –  Eumcoz Oct 15 '12 at 13:40

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.