Description
Configuration
I have 3 nodes, connected together using Tinc VPN, where I'd like to install HAproxy and have a VIP so that HAproxy is itself in a High Availability mode.
Here are the nodes details:
- Node 1 has IP Address 10.0.0.222/32 on interface vpn
- Node 2 has IP Address 10.0.0.13/32 on interface vpn
- Node 3 has IP Address 10.0.0.103/32 on interface vpn
To do so, I installed keepalived
on each machines.
I also enabled the following sysctl:
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
Node 1 has the following /etc/keepalived/keepalived.conf file:
global_defs {
enable_script_security
router_id node-1
}
vrrp_script haproxy-check {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
}
vrrp_instance haproxy-vip {
state MASTER
priority 150
interface vpn
virtual_router_id 1
advert_int 1
virtual_ipaddress {
10.0.0.1/32
}
track_script {
haproxy-check
}
}
Node 2 and 3 has the following /etc/keepalived/keepalived.conf file :
global_defs {
enable_script_security
router_id node-2 # Node 3 has "node-3" here.
}
vrrp_script haproxy-check {
script "/usr/bin/killall -0 haproxy"
interval 2
weight 2
}
vrrp_instance haproxy-vip {
state BACKUP
priority 100
interface vpn
virtual_router_id 1
advert_int 1
virtual_ipaddress {
10.0.0.1/32
}
track_script {
haproxy-check
}
}
When all the nodes are running keepalived
, Node 1 is the master, and the VIP 10.0.0.1
is well configured, the 2 other nodes ping it.
Node 1 logs
Logs when starting keepalived
:
Dec 5 14:07:53 node-1 systemd[1]: Starting Keepalive Daemon (LVS and VRRP)...
Dec 5 14:07:53 node-1 Keepalived[5870]: Starting Keepalived v1.3.2 (12/03,2016)
Dec 5 14:07:53 node-1 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Dec 5 14:07:53 node-1 Keepalived[5870]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Dec 5 14:07:53 node-1 Keepalived[5870]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 5 14:07:53 node-1 Keepalived[5871]: Starting Healthcheck child process, pid=5872
Dec 5 14:07:53 node-1 Keepalived_healthcheckers[5872]: Initializing ipvs
Dec 5 14:07:53 node-1 Keepalived_healthcheckers[5872]: Registering Kernel netlink reflector
Dec 5 14:07:53 node-1 Keepalived_healthcheckers[5872]: Registering Kernel netlink command channel
Dec 5 14:07:53 node-1 Keepalived_healthcheckers[5872]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 5 14:07:53 node-1 Keepalived[5871]: Starting VRRP child process, pid=5873
Dec 5 14:07:53 node-1 Keepalived_vrrp[5873]: Registering Kernel netlink reflector
Dec 5 14:07:53 node-1 Keepalived_vrrp[5873]: Registering Kernel netlink command channel
Dec 5 14:07:53 node-1 Keepalived_vrrp[5873]: Registering gratuitous ARP shared channel
Dec 5 14:07:53 node-1 Keepalived_vrrp[5873]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 5 14:07:53 node-1 Keepalived_healthcheckers[5872]: Using LinkWatch kernel netlink reflector...
Dec 5 14:07:53 node-1 Keepalived_vrrp[5873]: Using LinkWatch kernel netlink reflector...
Dec 5 14:07:53 node-1 Keepalived_vrrp[5873]: VRRP_Script(haproxy-check) succeeded
Dec 5 14:07:54 node-1 Keepalived_vrrp[5873]: VRRP_Instance(haproxy-vip) Transition to MASTER STATE
Dec 5 14:07:54 node-1 Keepalived_vrrp[5873]: VRRP_Instance(haproxy-vip) Changing effective priority from 150 to 152
Dec 5 14:07:55 node-1 Keepalived_vrrp[5873]: VRRP_Instance(haproxy-vip) Entering MASTER STATE
Dec 5 14:07:57 node-1 ntpd[946]: Listen normally on 45 vpn 10.0.0.1:123
Node 1's ip addr
:
vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.222/24 scope global vpn
valid_lft forever preferred_lft forever
inet 10.0.0.1/24 scope global secondary vpn
valid_lft forever preferred_lft forever
Node 2 and 3 logs
Dec 5 14:14:32 node-2 systemd[1]: Starting Keepalive Daemon (LVS and VRRP)...
Dec 5 14:14:32 node-2 Keepalived[13745]: Starting Keepalived v1.3.2 (12/03,2016)
Dec 5 14:14:32 node-2 Keepalived[13745]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Dec 5 14:14:32 node-2 Keepalived[13745]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 5 14:14:32 node-2 Keepalived[13746]: Starting Healthcheck child process, pid=13747
Dec 5 14:14:32 node-2 Keepalived_healthcheckers[13747]: Initializing ipvs
Dec 5 14:14:32 node-2 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Dec 5 14:14:32 node-2 Keepalived_healthcheckers[13747]: Registering Kernel netlink reflector
Dec 5 14:14:32 node-2 Keepalived_healthcheckers[13747]: Registering Kernel netlink command channel
Dec 5 14:14:32 node-2 Keepalived[13746]: Starting VRRP child process, pid=13748
Dec 5 14:14:32 node-2 Keepalived_healthcheckers[13747]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 5 14:14:32 node-2 Keepalived_vrrp[13748]: Registering Kernel netlink reflector
Dec 5 14:14:32 node-2 Keepalived_vrrp[13748]: Registering Kernel netlink command channel
Dec 5 14:14:32 node-2 Keepalived_vrrp[13748]: Registering gratuitous ARP shared channel
Dec 5 14:14:32 node-2 Keepalived_vrrp[13748]: Opening file '/etc/keepalived/keepalived.conf'.
Dec 5 14:14:32 node-2 Keepalived_healthcheckers[13747]: Using LinkWatch kernel netlink reflector...
Dec 5 14:14:32 node-2 Keepalived_vrrp[13748]: Using LinkWatch kernel netlink reflector...
Dec 5 14:14:32 node-2 Keepalived_vrrp[13748]: VRRP_Instance(haproxy-vip) Entering BACKUP STATE
Dec 5 14:14:32 node-2 Keepalived_vrrp[13748]: VRRP_Script(haproxy-check) succeeded
Dec 5 14:14:33 node-2 Keepalived_vrrp[13748]: VRRP_Instance(haproxy-vip) Changing effective priority from 100 to 102
Node 2 and 3's ip addr
:
Node 2
vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.13/24 scope global vpn
valid_lft forever preferred_lft forever
Node 3
vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.103/24 scope global vpn
valid_lft forever preferred_lft forever
The problem
But then, when I stop keepalived
on Node 1, Node 3 is elected as master, and registers the VIP, and only the Node 3 pings 10.0.0.1.
Node 1 logs
When stopping:
Dec 5 14:15:26 node-1 systemd[1]: Stopping Keepalive Daemon (LVS and VRRP)...
Dec 5 14:15:26 node-1 Keepalived[5871]: Stopping
Dec 5 14:15:26 node-1 Keepalived_healthcheckers[5872]: Stopped
Dec 5 14:15:26 node-1 Keepalived_vrrp[5873]: VRRP_Instance(haproxy-vip) sent 0 priority
Dec 5 14:15:27 node-1 Keepalived_vrrp[5873]: Stopped
Dec 5 14:15:27 node-1 Keepalived[5871]: Stopped Keepalived v1.3.2 (12/03,2016)
Dec 5 14:15:27 node-1 systemd[1]: Stopped Keepalive Daemon (LVS and VRRP).
Dec 5 14:15:28 node-1 ntpd[946]: Deleting interface #45 vpn, 10.0.0.1#123, interface stats: received=0, sent=0, dropped=0, active_time=451 secs
Node 1 ip addr
:
vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.222/24 scope global vpn
valid_lft forever preferred_lft forever
Node 2 logs
Dec 5 14:15:27 node-2 Keepalived_vrrp[13748]: VRRP_Instance(haproxy-vip) Transition to MASTER STATE
Dec 5 14:15:27 node-2 Keepalived_vrrp[13748]: VRRP_Instance(haproxy-vip) Received advert with higher priority 102, ours 102
Dec 5 14:15:27 node-2 Keepalived_vrrp[13748]: VRRP_Instance(haproxy-vip) Entering BACKUP STATE
Node 2 ip addr
:
vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.13/24 scope global vpn
valid_lft forever preferred_lft forever
Node 3 logs
Dec 5 14:15:27 node-3 Keepalived_vrrp[31252]: VRRP_Instance(haproxy-vip) Transition to MASTER STATE
Dec 5 14:15:27 node-3 Keepalived_vrrp[31252]: VRRP_Instance(haproxy-vip) Received advert with lower priority 102, ours 102, forcing new election
Dec 5 14:15:28 node-3 Keepalived_vrrp[31252]: VRRP_Instance(haproxy-vip) Entering MASTER STATE
Dec 5 14:15:29 node-3 ntpd[27734]: Listen normally on 36 vpn 10.0.0.1:123
Node 3 ip addr
:
vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.103/24 scope global vpn
valid_lft forever preferred_lft forever
inet 10.0.0.1/24 scope global secondary vpn
valid_lft forever preferred_lft forever
Some more details
traceroute
I used traceroute
in order to try to get some more information about the issue.
When all nodes are running keepalived
and pinging the VIP works everywhere, traceroute
shows that for all the nodes :
$ traceroute 10.0.0.1
traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 60 byte packets
1 10.0.0.1 (10.0.0.1) 0.094 ms 0.030 ms 0.019 ms
When keepalived
is stopped on Node 1, and Node 3 got elected, Node 1 can't figure out where is the VIP:
$ traceroute 10.0.0.1
traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 60 byte packets
1 * * *
2 * * *
...
29 * * *
30 * * *
Node 2 expects Node 1 to have the VIP:
$ traceroute 10.0.0.1
traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 60 byte packets
1 10.0.0.222 (10.0.0.222) 0.791 ms 0.962 ms 1.080 ms
2 * * *
3 * * *
...
And Node 3 has the VIP, so it works.
Tinc DeviceType
I read some mail archive where it was proposed to use the DeviceType = tap
in the Tinc config in order to get the ARP paquets transmitted (as far as I understood), but it didn't helped.
Actually I'm not sure Tinc is the root cause, as the election happens.
Try without Tinc
I changed the keepalived
config so that it uses the public internet interface, using unicast.
I have added the following block to each keepalived config on each nodes (here is for node-1
):
unicast_src_ip XXX.XXX.XXX.XXX # node's public IP address
unicast_peer {
XXX.XXX.XXX.XXX # other node's public IP address
XXX.XXX.XXX.XXX # other node's public IP address
}
But the behaviour is exactly the same as describe above, so Tinc shouldn't be related.
Request
Could anyone help me to figure out what is going wrong and solved this issue so that when a new election is made, the nodes finds the VIP on its new place?