0

Description

Configuration

I have 3 nodes, connected together using Tinc VPN, where I'd like to install HAproxy and have a VIP so that HAproxy is itself in a High Availability mode.

Here are the nodes details:

  • Node 1 has IP Address 10.0.0.222/32 on interface vpn
  • Node 2 has IP Address 10.0.0.13/32 on interface vpn
  • Node 3 has IP Address 10.0.0.103/32 on interface vpn

To do so, I installed keepalived on each machines.

I also enabled the following sysctl:

net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1

Node 1 has the following /etc/keepalived/keepalived.conf file:

global_defs {
  enable_script_security
  router_id node-1
}

vrrp_script haproxy-check {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance haproxy-vip {
    state MASTER
    priority 150
    interface vpn
    virtual_router_id 1
    advert_int 1

    virtual_ipaddress {
        10.0.0.1/32
    }

    track_script {
        haproxy-check
    }
}

Node 2 and 3 has the following /etc/keepalived/keepalived.conf file :

global_defs {
  enable_script_security
  router_id node-2 # Node 3 has "node-3" here.
}

vrrp_script haproxy-check {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance haproxy-vip {
    state BACKUP
    priority 100
    interface vpn
    virtual_router_id 1
    advert_int 1

    virtual_ipaddress {
        10.0.0.1/32
    }

    track_script {
        haproxy-check
    }
}

When all the nodes are running keepalived, Node 1 is the master, and the VIP 10.0.0.1 is well configured, the 2 other nodes ping it.

Node 1 logs

Logs when starting keepalived:

Dec  5 14:07:53 node-1 systemd[1]: Starting Keepalive Daemon (LVS and VRRP)...
Dec  5 14:07:53 node-1 Keepalived[5870]: Starting Keepalived v1.3.2 (12/03,2016)
Dec  5 14:07:53 node-1 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Dec  5 14:07:53 node-1 Keepalived[5870]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Dec  5 14:07:53 node-1 Keepalived[5870]: Opening file '/etc/keepalived/keepalived.conf'.
Dec  5 14:07:53 node-1 Keepalived[5871]: Starting Healthcheck child process, pid=5872
Dec  5 14:07:53 node-1 Keepalived_healthcheckers[5872]: Initializing ipvs
Dec  5 14:07:53 node-1 Keepalived_healthcheckers[5872]: Registering Kernel netlink reflector
Dec  5 14:07:53 node-1 Keepalived_healthcheckers[5872]: Registering Kernel netlink command channel
Dec  5 14:07:53 node-1 Keepalived_healthcheckers[5872]: Opening file '/etc/keepalived/keepalived.conf'.
Dec  5 14:07:53 node-1 Keepalived[5871]: Starting VRRP child process, pid=5873
Dec  5 14:07:53 node-1 Keepalived_vrrp[5873]: Registering Kernel netlink reflector
Dec  5 14:07:53 node-1 Keepalived_vrrp[5873]: Registering Kernel netlink command channel
Dec  5 14:07:53 node-1 Keepalived_vrrp[5873]: Registering gratuitous ARP shared channel
Dec  5 14:07:53 node-1 Keepalived_vrrp[5873]: Opening file '/etc/keepalived/keepalived.conf'.
Dec  5 14:07:53 node-1 Keepalived_healthcheckers[5872]: Using LinkWatch kernel netlink reflector...
Dec  5 14:07:53 node-1 Keepalived_vrrp[5873]: Using LinkWatch kernel netlink reflector...
Dec  5 14:07:53 node-1 Keepalived_vrrp[5873]: VRRP_Script(haproxy-check) succeeded
Dec  5 14:07:54 node-1 Keepalived_vrrp[5873]: VRRP_Instance(haproxy-vip) Transition to MASTER STATE
Dec  5 14:07:54 node-1 Keepalived_vrrp[5873]: VRRP_Instance(haproxy-vip) Changing effective priority from 150 to 152
Dec  5 14:07:55 node-1 Keepalived_vrrp[5873]: VRRP_Instance(haproxy-vip) Entering MASTER STATE
Dec  5 14:07:57 node-1 ntpd[946]: Listen normally on 45 vpn 10.0.0.1:123

Node 1's ip addr:

vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.222/24 scope global vpn
   valid_lft forever preferred_lft forever
inet 10.0.0.1/24 scope global secondary vpn
   valid_lft forever preferred_lft forever

Node 2 and 3 logs

Dec  5 14:14:32 node-2 systemd[1]: Starting Keepalive Daemon (LVS and VRRP)...
Dec  5 14:14:32 node-2 Keepalived[13745]: Starting Keepalived v1.3.2 (12/03,2016)
Dec  5 14:14:32 node-2 Keepalived[13745]: WARNING - default user 'keepalived_script' for script execution does not exist - please create.
Dec  5 14:14:32 node-2 Keepalived[13745]: Opening file '/etc/keepalived/keepalived.conf'.
Dec  5 14:14:32 node-2 Keepalived[13746]: Starting Healthcheck child process, pid=13747
Dec  5 14:14:32 node-2 Keepalived_healthcheckers[13747]: Initializing ipvs
Dec  5 14:14:32 node-2 systemd[1]: Started Keepalive Daemon (LVS and VRRP).
Dec  5 14:14:32 node-2 Keepalived_healthcheckers[13747]: Registering Kernel netlink reflector
Dec  5 14:14:32 node-2 Keepalived_healthcheckers[13747]: Registering Kernel netlink command channel
Dec  5 14:14:32 node-2 Keepalived[13746]: Starting VRRP child process, pid=13748
Dec  5 14:14:32 node-2 Keepalived_healthcheckers[13747]: Opening file '/etc/keepalived/keepalived.conf'.
Dec  5 14:14:32 node-2 Keepalived_vrrp[13748]: Registering Kernel netlink reflector
Dec  5 14:14:32 node-2 Keepalived_vrrp[13748]: Registering Kernel netlink command channel
Dec  5 14:14:32 node-2 Keepalived_vrrp[13748]: Registering gratuitous ARP shared channel
Dec  5 14:14:32 node-2 Keepalived_vrrp[13748]: Opening file '/etc/keepalived/keepalived.conf'.
Dec  5 14:14:32 node-2 Keepalived_healthcheckers[13747]: Using LinkWatch kernel netlink reflector...
Dec  5 14:14:32 node-2 Keepalived_vrrp[13748]: Using LinkWatch kernel netlink reflector...
Dec  5 14:14:32 node-2 Keepalived_vrrp[13748]: VRRP_Instance(haproxy-vip) Entering BACKUP STATE
Dec  5 14:14:32 node-2 Keepalived_vrrp[13748]: VRRP_Script(haproxy-check) succeeded
Dec  5 14:14:33 node-2 Keepalived_vrrp[13748]: VRRP_Instance(haproxy-vip) Changing effective priority from 100 to 102

Node 2 and 3's ip addr:

Node 2

vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.13/24 scope global vpn
   valid_lft forever preferred_lft forever

Node 3

vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.103/24 scope global vpn
   valid_lft forever preferred_lft forever

The problem

But then, when I stop keepalived on Node 1, Node 3 is elected as master, and registers the VIP, and only the Node 3 pings 10.0.0.1.

Node 1 logs

When stopping:

Dec  5 14:15:26 node-1 systemd[1]: Stopping Keepalive Daemon (LVS and VRRP)...
Dec  5 14:15:26 node-1 Keepalived[5871]: Stopping
Dec  5 14:15:26 node-1 Keepalived_healthcheckers[5872]: Stopped
Dec  5 14:15:26 node-1 Keepalived_vrrp[5873]: VRRP_Instance(haproxy-vip) sent 0 priority
Dec  5 14:15:27 node-1 Keepalived_vrrp[5873]: Stopped
Dec  5 14:15:27 node-1 Keepalived[5871]: Stopped Keepalived v1.3.2 (12/03,2016)
Dec  5 14:15:27 node-1 systemd[1]: Stopped Keepalive Daemon (LVS and VRRP).
Dec  5 14:15:28 node-1 ntpd[946]: Deleting interface #45 vpn, 10.0.0.1#123, interface stats: received=0, sent=0, dropped=0, active_time=451 secs

Node 1 ip addr:

vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.222/24 scope global vpn
   valid_lft forever preferred_lft forever

Node 2 logs

Dec  5 14:15:27 node-2 Keepalived_vrrp[13748]: VRRP_Instance(haproxy-vip) Transition to MASTER STATE
Dec  5 14:15:27 node-2 Keepalived_vrrp[13748]: VRRP_Instance(haproxy-vip) Received advert with higher priority 102, ours 102
Dec  5 14:15:27 node-2 Keepalived_vrrp[13748]: VRRP_Instance(haproxy-vip) Entering BACKUP STATE

Node 2 ip addr:

vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.13/24 scope global vpn
   valid_lft forever preferred_lft forever

Node 3 logs

Dec  5 14:15:27 node-3 Keepalived_vrrp[31252]: VRRP_Instance(haproxy-vip) Transition to MASTER STATE
Dec  5 14:15:27 node-3 Keepalived_vrrp[31252]: VRRP_Instance(haproxy-vip) Received advert with lower priority 102, ours 102, forcing new election
Dec  5 14:15:28 node-3 Keepalived_vrrp[31252]: VRRP_Instance(haproxy-vip) Entering MASTER STATE
Dec  5 14:15:29 node-3 ntpd[27734]: Listen normally on 36 vpn 10.0.0.1:123

Node 3 ip addr:

vpn: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500
link/none
inet 10.0.0.103/24 scope global vpn
   valid_lft forever preferred_lft forever
inet 10.0.0.1/24 scope global secondary vpn
   valid_lft forever preferred_lft forever

Some more details

traceroute

I used traceroute in order to try to get some more information about the issue.

When all nodes are running keepalived and pinging the VIP works everywhere, traceroute shows that for all the nodes :

$ traceroute 10.0.0.1
traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 60 byte packets
 1  10.0.0.1 (10.0.0.1)  0.094 ms  0.030 ms  0.019 ms

When keepalived is stopped on Node 1, and Node 3 got elected, Node 1 can't figure out where is the VIP:

$ traceroute 10.0.0.1
traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 ...
 29  * * *
 30  * * *

Node 2 expects Node 1 to have the VIP:

$ traceroute 10.0.0.1
traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 60 byte packets
 1  10.0.0.222 (10.0.0.222)  0.791 ms  0.962 ms  1.080 ms
 2  * * *
 3  * * *
 ...

And Node 3 has the VIP, so it works.

Tinc DeviceType

I read some mail archive where it was proposed to use the DeviceType = tap in the Tinc config in order to get the ARP paquets transmitted (as far as I understood), but it didn't helped.

Actually I'm not sure Tinc is the root cause, as the election happens.

Try without Tinc

I changed the keepalived config so that it uses the public internet interface, using unicast.

I have added the following block to each keepalived config on each nodes (here is for node-1):

    unicast_src_ip XXX.XXX.XXX.XXX # node's public IP address
    unicast_peer {
        XXX.XXX.XXX.XXX # other node's public IP address
        XXX.XXX.XXX.XXX # other node's public IP address
    }

But the behaviour is exactly the same as describe above, so Tinc shouldn't be related.

Request

Could anyone help me to figure out what is going wrong and solved this issue so that when a new election is made, the nodes finds the VIP on its new place?

1 Answer 1

1

I just solved a similar problem by adding Mode = switch to my tinc.conf.

The issue I faced was similar to what you describe; keepalived would transition the virtual ip I setup (pointing to a simple nginx server) between my 3 nodes as expected. However, the only node able to reach the service was the elected MASTER. This was due to the routing table being built statically, from the host configuration files, rather than from ARP data.

I do find it odd that your attempt without tinc failed. When I changed my configs to run over my local network, which has a router, keepalived and haproxy functioned as expected and the vip was visible in the router's ARP table. Are you sure you changed both the haproxy and keepalived configs for you local test?

Best of luck!

References:

  1. https://www.tinc-vpn.org/documentation/Main-configuration-variables.html
  2. https://www.tinc-vpn.org/pipermail/tinc/2010-February/002191.html
3
  • Thank you for your answer. I'll retry my test without Tinc, and also try with the Mode = switch in Tinc, and I'll let you know.
    – ZedTuX
    Feb 17, 2020 at 8:11
  • I tried again to change the keepalived config so that it uses the public internet addresses instead of Tinc, I have updated the question's part in order to show how I did it, but I still have the same behavior. Why should I change the haproxy config? I'm just trying to ping the 10.0.0.1 IP address.
    – ZedTuX
    Mar 25, 2020 at 5:14
  • Whatever, I've updated the Tinc Mode to switch and all works fine, thank you !
    – ZedTuX
    Mar 25, 2020 at 5:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .