I am using ldirectord for loadbalancing two HTTP servers. On the load balancer box i have the following network configuration:
eth0 is used for internal purposes and does not take part in load balancing at all. On eth1 I have configured the public IP for the machine (for accessses from other internal networks) and the VIP of the load balancer. eth2 is used for accessing the real servers, which lie in different subnetwork and are only accessible through IPs from the same subnet.
The details are as follows:
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet 127.0.0.2/8 brd 127.255.255.255 scope host secondary lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:50:56:a5:77:ae brd ff:ff:ff:ff:ff:ff
inet 192.168.8.216/22 brd 192.168.11.255 scope global eth0
inet6 fe80::250:56ff:fea5:77ae/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:50:56:a5:77:af brd ff:ff:ff:ff:ff:ff
inet 172.22.9.100/22 brd 172.22.11.255 scope global eth1:1
inet 172.22.8.213/22 brd 172.22.11.255 scope global secondary eth1
inet6 fe80::250:56ff:fea5:77af/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:50:56:a5:77:b0 brd ff:ff:ff:ff:ff:ff
inet 172.22.1.130/24 brd 172.22.1.255 scope global eth2
inet6 fe80::250:56ff:fea5:77b0/64 scope link
valid_lft forever preferred_lft forever
Here is the configuration of my NAT load balancing:
# cat /etc/ha.d/ldirectord.cf
autoreload = yes
quiescent = yes
checkinterval = 10
negotiatetimeout = 10
checktimeout = 5
emailalert = "[email protected]"
emailalertfreq = 60
failurecount = 3
virtual = 172.22.9.100:80
checktimeout = 10
checktype = negotiate
failurecount = 2
negotiatetimeout = 10
protocol = tcp
quiescent = yes
real = 172.22.1.133:80 masq 1024
real = 172.22.1.134:80 masq 1024
request = "alive.htm"
receive = "I am alive"
scheduler = wrr
service = http
persistent = 5
I have also two additional iptables rules to make the NATing work as expected:
# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
SNAT all -- anywhere anywhere to:172.22.9.100
SNAT all -- anywhere anywhere to:172.22.1.130
Now, the problem is, that the alive checking does not work as expected. Although the servers are online (I can ping them and all), the ldirector does not see it that way and marks them as down.
If I use ping checking everything is working fine, but we need the negotiate for more comprehensive service checks (server cna be pingable but http not working, etc).
I tried to download the alive.htm page from the command line on the load balancer box and failed.
# wget http://172.22.1.133/alive.htm
--2013-04-11 09:52:44-- http://172.22.1.133/alive.htm
Connecting to 172.22.1.133:80... failed: Connection timed out.
Retrying.
First I thought that the problem is with the iptables rules. I removed them but still could not download the alive page with wget (and curl and w3m for that matter). i tried wgetting it from two other machines, both from different subnetworks and it worked fine! I realized something is not OK with the load balancer box.
I then shut down the ldirectord and tried the wget again. This time it worked.
So it seems to me that the ldirector somehow stops itself from accessing the alive page on the real servers? That seems strange and unlogical. So what can I be doing wrong?
The issues is no caused by the iptables rules, because when I remove them the alive.htm page is still not available.
The real servers are both accessible from the load balancer (pingable). Do I have some configuration issues of ldirectord?
How can I track exactly what the ldirectord is doing during the alive test?
And generally, can someone shed some light or give me some ideas what could be wrong or what I could be doing wrong.
Thanks in advance :)