I have three back end servers load balanced using HAProxy.
When all 3 of those are up i see each servers Id (1,2,3) consecutively in the HTTPS response. IE 1,2,3,1,2,3. As expected.
However, if i take one back end server down i then only see one server id repeatedly.
I.e if I take server 1 down I then only see server id 2 (2,2,2,2...) when I should see server id's 2,3 (2,3,2,3,2,3.....)
I have tried both leastconn and round robin balancing and both exhibit the same behaviour
Why does HAProxy behave this way? How do I alter my HAProxy config to load balance across the remaining alive back end servers?
HA proxy version is 1.6.9. Server for HA proxy is ubuntu 14.04.
The HTTP being served is an API that returns some JSON over HTTPS. Its a restful API so there is no session persistence required or wanted.
The configuration for haproxy is below.
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 3072
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH$
ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
option dontlog-normal
option redispatch
retries 3
maxconn 3072
timeout connect 5000
timeout client 10000
timeout server 10000
frontend apis-frontend
bind x.x.x.x:443 ssl verify none crt /etc/haproxy/xxx
mode http
option forwardfor
default_backend apis
backend apis
balance leastconn
mode http
option httpchk GET /
server 1 x.x.x.x:443 ssl verify none check
server 2 x.x.x.x:443 ssl verify none check
server 3 x.x.x.x:443 ssl verify none check
listen stats
bind *:8181
mode http
stats enable
stats uri /
stats realm Haproxy\ Statistics
stats auth xx
Edit: added stats.
pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,
apis-frontend,FRONTEND,,,0,2,3072,24,9118,10021,0,0,19,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,4,,,,0,24,0,19,0,0,,0,6,43,,,0,0,0,0,,,,,,,,
apis,mt-wol-vlx-vps-01,0,0,0,1,,12,4863,3240,,0,,0,0,0,0,UP,1,1,0,1,0,12097,0,,1,3,1,,12,,2,0,,2,L7OK,200,36,0,12,0,0,0,0,0,,,,0,0,,,,,39,OK,,0,1,1,29,
apis,mt-lon-vlx-vps-01,0,0,0,1,,12,4255,3228,,0,,0,0,0,0,UP,1,1,0,0,0,12097,0,,1,3,2,,12,,2,0,,2,L7OK,200,23,0,12,0,0,0,0,0,,,,0,0,,,,,39,OK,,0,1,1,5,
apis,mt-cov-uks-vps-01,0,0,0,0,,0,0,0,,0,,0,0,0,0,DOWN,1,1,0,1,1,12094,12094,,1,3,3,,0,,2,0,,0,L4TOUT,,2001,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
apis,BACKEND,0,0,0,1,308,24,9118,6468,0,0,,0,0,0,0,UP,2,2,0,,0,12097,0,,1,3,0,,24,,1,0,,3,,,,0,24,0,0,0,0,,,,,0,0,0,0,0,0,39,,,0,1,1,33,
stats,FRONTEND,,,1,4,3072,10,3811,169181,0,0,5,,,,,OPEN,,,,,,,,,1,4,0,,,,0,1,0,4,,,,0,9,0,5,0,0,,1,3,15,,,0,0,0,0,,,,,,,,
stats,BACKEND,0,0,0,0,308,0,3811,169181,0,0,,0,0,0,0,UP,0,0,0,,0,12097,0,,1,4,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,0,,,0,0,0,18,
edit: load balancing behaviour in different browsers
After a request to test from a different ip. I noticed the second ip load balanced as expected. I then tried from the first IP but using different browsers edge and firefox. Both edge and firefox load balance as expected.
However the original browser chrome still stick to one back end server. Even after closing all chrome windows and restarting chrome
