0

I currently have HAProxy 1.5.4 setup using keepalived for HA on centos 7 using IPtables to create a transparent load balancer to load balance a number of cisco WSA proxy servers for internet filtering on an infrastructure of around 15,000 users running on vmware and cisco UCS with one GB nic.

All has been running well until the load was increased, at first i realised the maxconn on the frontend was set at default and has now been changed to 5000.

Being new to HAProxy and having only set it up following a guide for our exchange environment could anyone point out any obvious issues in my config below and any performance improvements I could make.

edit Users have been reporting a lack of internet connectivity which usually happens when the current sessions reaches around 2,500-3000.

HAProxy Config:

global
    daemon
    log /dev/log local4
    maxconn 40000
    ulimit-n 81000

defaults
    log global
    timeout connect 4000
    timeout client 42000
    timeout server 43000
    mode http

frontend http-in
    bind *:80
    maxconn 5000
    default_backend backend_servers
    option  forwardfor
    option http-server-close

backend backend_servers
    balance leastconn
    stick-table type ip size 10240k expire 10m
    stick on src
    server  wsa01 10.80.10.111:80 check inter 30000 fall 5
    server  wsa02 10.80.10.112:80 check inter 30000 fall 5
    server  wsa03 10.80.10.113:80 check inter 30000 fall 5
    source 0.0.0.0 usesrc clientip

listen stats *:7000
   stats enable
   stats hide-version
   stats refresh 30s
   stats show-node
   stats uri     /
   stats auth    admin:REDACTED

IPtables Config:

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DIVERT - [0:0]
-A PREROUTING -p tcp -m socket -j DIVERT
-A DIVERT -j MARK --set-xmark 0x6f/0xffffffff
-A DIVERT -j ACCEPT
COMMIT

Added to rc.local for iptables:

iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 111
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 111 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100

Keepalived Config:

! Configuration File for keepalived

global_defs {
notification_email {
    REDACTED@EMAIL
}
notification_email_from REDACTED@EMAIL
smtp_server 10.80.11.200
smtp_connect_timeout 30
}

vrrp_script chk_haproxy {
script "killall -0 haproxy" # this will check if the haproxy is up
interval 45 # check every 5 seconds
weight 2 # add 2 points of priority if OK
}

vrrp_instance VI_1 {
interface ens32
state MASTER # or "BACKUP" on backup
priority 101 # 101 on master, 100 on backup
virtual_router_id 60

smtp_alert # Activate SMTP notifications, you can remove this if you dont want alerts

authentication {
    auth_type PASS
    auth_pass REDACTED
 }

virtual_ipaddress {
    10.80.10.200
}
track_script {
    chk_haproxy
}
}

Many Thanks in advance for any assistance...

Rebus

  • What kind of performance issues are you seeing? – GregL Jul 15 '15 at 15:12
  • What kind of load is the server seeing? Is it memor-bound or CPU-bound? As Justin has answered below, you might be able to just increase the process count and be on your way. – GregL Jul 15 '15 at 15:26
  • I'm currently seeing around 50-60% CPU usage, about 10% memory usage and around 25% bandwidth usage, i was looking at nbproc but as the single core cpu is around 60% I am unsure if it will make a difference. – Rebus Jul 15 '15 at 16:04
  • When you get reports of lack of connectivity, are the connections being queued in HAproxy, or are they simply not being openned at all? – GregL Jul 15 '15 at 16:51
  • it seems they are just not being opened as looking at the stats page for the front end queue its just blank and the server queue is all zeroes (17day up time), one thing i have found odd is that the back end servers (ironport) seem to be holding on to sessions as typical current connections is 6-8k but i have one on 34k when behind haproxy. – Rebus Jul 20 '15 at 10:01
0

You can add nbproc N to Global, where N is the number of cores on your server. This will get HAproxy to use all the cores. You can allocate groups of processors to different frontends and backends to, for example, prioritise a frontend that does SSL.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.