Bond active backup Linux

Question

i am trying to set up a bond on my 4.4 proxmox server.

i chose active-backup because the bond will be linked to a linux bridge which will not use Spanningtree Protocol.

the bond(bond1) will have 2 slaves : Bond0 which is an LACP of 3 links(eth0,eth1,eth2) to 1 switch and eth3 to an other switch.

My problem is that when i unplug Bond0 (primary interface of Bond1) i unplug all the cables from (eth0 eth1 eth2)

Bond1 keep bond0 as active link and as effect: the server is unreachable from the local network.

i think this is because Bond0 remain UP even when its 3 slaves are down.

because When i do ifconfig bond0 down.

Bond1 use the other link (eth3) for connection. and the server is back on the local network.

that's the way it is supposed to work.

So i tried multiple parameters in bond0, to force it to have a DOWN state when eth0 eth1 and eth2 are DOWN.

I couldn't find out.

i even used the post-down,post-up,.... parameters on eth0 eth1 and eth2 in /etc/network/interface, to trigger a script that do ifdown bond0 when these ports are DOWN and ifup bond0 when one of them is back working.

But this isn't working, i tried unplugging bond0. bond1 still use bond0 as active link. The script isn't running.

At this point i wanted to pull out my hair....

The script is not triggered because. they are executed only when i do ifdown eth0 ifdown eth1 or ifdown eth2...

(post-down,post-up): now i know that these parameters shot a command only when we run ifdown or ifup, but not when we unplug the plug. (seriousely what's the point of this fonctionality).

i want to execute an action when an interface is down and it don't work when i unplug the ethernet cable.....

that's not what i want at all.

i restarted between changes.

i'm kind of desperate right now, i spent 9 hours on this.

so i am asking you your help.

i found a debian package "ifplugd" which will do actions when interfaces are unplugged. I will try it tomorow

I don't really know what's the best practice here.

I am listening of any advices. Thank you.

update

Here is /etc/network/interface:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
        post-down do_i_bring_bond0_down.sh
        post-up do_i_bring_bond0_up.sh

auto eth1
iface eth1 inet manual
        post-down do_i_bring_bond0_down.sh
        post-up do_i_bring_bond0_up.sh

auto eth2
iface eth2 inet manual
        post-down do_i_bring_bond0_down.sh
        post-up do_i_bring_bond0_up.sh

iface eth3 inet manual


auto bond0
iface bond0 inet manual
        slaves eth0 eth1 eth2
        bond_miimon 100
        bond_mode 802.3ad
        bond_xmit_hash_policy layer3+4
        bond_min_links 1

auto bond1
iface bond1 inet manual
        slaves bond0 eth3
        bond_miimon 100
        bond_mode active-backup
        bond_primary bond0
        bond_updelay 100
        bond_downdelay 100

auto vmbr0
iface vmbr0 inet static
        address  10.4.2.4
        netmask  255.255.255.192
        gateway  10.4.2.1
        bridge_ports bond1.20
        bridge_stp off
        bridge_fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge_ports bond1
        bridge_stp off
        bridge_fd 0
        bridge_vlan_aware yes

/proc/net/bonding/bond1

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: bond0 (primary_reselect always)
Currently Active Slave: bond0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 100
Down Delay (ms): 100

Slave Interface: bond0
MII Status: up
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: c8:1f:66:c2:dc:3b
Slave queue ID: 0

Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:15:17:3a:a9:bf
Slave queue ID: 0

/proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: c8:1f:66:c2:dc:3b
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 3
        Actor Key: 9
        Partner Key: 67
        Partner Mac Address: 80:2a:a8:df:42:b1

Slave Interface: eth0
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 1
Permanent HW addr: c8:1f:66:c2:dc:3b
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: c8:1f:66:c2:dc:3b
    port key: 0
    port priority: 255
    port number: 1
    port state: 61
details partner lacp pdu:
    system priority: 32768
    system mac address: 80:2a:a8:df:42:b1
    oper key: 67
    port priority: 128
    port number: 8
    port state: 61

Slave Interface: eth1
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 1
Permanent HW addr: c8:1f:66:c2:dc:3c
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system mac address: c8:1f:66:c2:dc:3b
    port key: 0
    port priority: 255
    port number: 2
    port state: 61
details partner lacp pdu:
    system priority: 32768
    system mac address: 80:2a:a8:df:42:b1
    oper key: 67
    port priority: 128
    port number: 7
    port state: 61

Slave Interface: eth2
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 1
Permanent HW addr: 00:15:17:3a:a9:be
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
    system priority: 65535
    system mac address: c8:1f:66:c2:dc:3b
    port key: 0
    port priority: 255
    port number: 3
    port state: 61
details partner lacp pdu:
    system priority: 32768
    system mac address: 80:2a:a8:df:42:b1
    oper key: 67
    port priority: 128
    port number: 9
    port state: 61

Answer 1

You cannot bond bonds, this configuration is wrong.

See my other answer for a way to achieve "failover between bonds" solely with LACP, which is the correct configuration: https://unix.stackexchange.com/questions/82569/bonds-vs-aggregators/172232#172232

Answer 2

The problem seems to be this: "Min links: 0". From the bonding documentation for the min_links parameter

min_links

This allows setting the minimum number of member ports that must be up (link-up state) before marking the bond device as up (carrier on)

The default value is 0. This will cause carrier to be asserted (for 802.3ad mode) whenever there is an active aggregator, regardless of the number of available links in that aggregator.

However, this part is somewhat confusing:

Note that, because an aggregator cannot be active without at least one available link, setting this option to 0 or to 1 has the exact same effect.

In your bond0 example all three eth interfaces have MII status (link-state) of down but the aggragator they are in (2) is active which seems to contradict the documentation. Let's look at the source.

The code for bond_3ad_set_carrier() calls __agg_active_ports() which simply checks the value of the "is_enabled" flag of each dev and has seemingly nothing to do with link status.

This patch seems to indicate that this is the case:

For now, the patch uses number of active slaves to compare against min_links to know when to on/off bond carrier. Suggested setting of min_links is 1, rather than the default of zero. Using min_links=1 says that atleast 1 slave must be active within bond for bond to be carrier on.

From what I can tell the patch hasn't been integrated into mainline or maybe it has in a modified form I could not discern.

Try setting min_links to 1 or 2 and see what happens. If that doesn't work you could switch from miimon to arp monitoring (setting arp_interval apr_ip_target, etc.)

P.S. If you end up using LACP I'd change lacp_rate from the default (slow LACPDUs every 30 seconds - i.e. time to detect failure is 30 seconds) to fast or 1 (Request partner to transmit LACPDUs every 1 second).

Edit: Looking into the code more has made me question my answer. The bond_3ad_handle_link_change() function should disable a dev when the link status changes. Maybe you didn't wait 30+ seconds for the LACP state machine to work? I'll leave the answer nontheless because I spent so much time on it. :)