I am trying to setup HAProxy to load balance a group of Thrift servers. For some reason the HAProxy stats page says the servers are all down.
Here is the current HAProxy config I am trying.
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 60000
user haproxy
group haproxy
defaults
log global
option dontlognull
option redispatch
retries 3
maxconn 2000
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen stats :5000
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth user:pass
listen metrix :5002
mode tcp
option tcplog
balance leastconn
server m1 127.0.0.1:9000 check
server m2 127.0.0.1:9001 check
server m3 127.0.0.1:9002 check
server m4 127.0.0.1:9003 check
server m5 127.0.0.1:9004 check
server m6 127.0.0.1:9005 check
server m7 127.0.0.1:9006 check
server m8 127.0.0.1:9007 check
One other things. The Thrift servers are running in Supervisor which I have noticed some weirdness with. But I have tried just running the Thrift server not in supervisor and it still doesn't work.
I have tried all of the Thrift servers (TSimpleServer, TNonBlockingServer, TThreadedServer) all of them have the same problem.
Update
Running tcpdump port 9000
15:12:31.878502 IP ip-00.00.00.00.ec2.internal.cslistener > ip-11.11.11.11.ec2.internal.36206: Flags [R.], seq 0, ack 3433673377, win 0, length 0
15:12:33.878425 IP ip-11.11.11.11.ec2.internal.36207 > ip-00.00.00.00.ec2.internal.cslistener: Flags [S], seq 3459211721, win 5840, options [mss 1460,sackOK,TS val 440815982 ecr 0,nop,wscale 10], length 0
Where 00.00.00.00
is the server running the Thrift server and 11.11.11.11
is the server running HAProxy.
These 2 lines repeat continuously.
Running netstat -tlnp
contains the following:
tcp 0 0 127.0.0.1:9000 0.0.0.0:* LISTEN 19472/python
So the Thrift server is listening on the right port.
tcpdump
on127.0.0.1
port 9000?15:12:31.878502 IP ip-00.00.00.00.ec2.internal.cslistener > ip-11.11.11.11.ec2.internal.36206: Flags [R.], seq 0, ack 3433673377, win 0, length 0 15:12:33.878425 IP ip-11.11.11.11.ec2.internal.36207 > ip-00.00.00.00.ec2.internal.cslistener: Flags [S], seq 3459211721, win 5840, options [mss 1460,sackOK,TS val 440815982 ecr 0,nop,wscale 10], length 0
Where00.00.00.00
is the server running the Thrift server and11.11.11.11
is the server running HAProxy.