Bonding: works only for download

Question

I would like to install bonding with 4 links with mode 4. but only "download/receiving" works with bondig. for transmitting the system chooses one link.

ifconfig

bond0     Link encap:Ethernet  HWaddr 90:E2:BA:0F:76:B4
          inet addr:ip  Bcast:ip  Mask:255.255.255.248
          inet6 addr: fe80::92e2:baff:fe0f:76b4/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:239187413 errors:0 dropped:10944 overruns:0 frame:0
          TX packets:536902370 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:14688536197 (13.6 GiB)  TX bytes:799521192901 (744.6 GiB)

eth2      Link encap:Ethernet  HWaddr 90:E2:BA:0F:76:B4
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:54969488 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2537 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3374778591 (3.1 GiB)  TX bytes:314290 (306.9 KiB)

eth3      Link encap:Ethernet  HWaddr 90:E2:BA:0F:76:B4
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:64935805 errors:0 dropped:1 overruns:0 frame:0
          TX packets:2532 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3993499746 (3.7 GiB)  TX bytes:313968 (306.6 KiB)

eth4      Link encap:Ethernet  HWaddr 90:E2:BA:0F:76:B4
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:57352105 errors:0 dropped:2 overruns:0 frame:0
          TX packets:536894778 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3524236530 (3.2 GiB)  TX bytes:799520265627 (744.6 GiB)

eth5      Link encap:Ethernet  HWaddr 90:E2:BA:0F:76:B4
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:61930025 errors:0 dropped:3 overruns:0 frame:0
          TX packets:2540 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3796021948 (3.5 GiB)  TX bytes:314274 (306.9 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:62 errors:0 dropped:0 overruns:0 frame:0
          TX packets:62 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5320 (5.1 KiB)  TX bytes:5320 (5.1 KiB)

those are my configs:

DEVICE="eth2"
BOOTPROTO="none"
MASTER=bond0
SLAVE=yes
USERCTL=no
NM_CONTROLLED="no"
ONBOOT="yes"


DEVICE="eth3"
BOOTPROTO="none"
MASTER=bond0
SLAVE=yes
USERCTL=no
NM_CONTROLLED="no"
ONBOOT="yes"



DEVICE="eth4"
BOOTPROTO="none"
MASTER=bond0
SLAVE=yes
USERCTL=no
NM_CONTROLLED="no"
ONBOOT="yes"


DEVICE="eth5"
BOOTPROTO="none"
MASTER=bond0
SLAVE=yes
USERCTL=no
NM_CONTROLLED="no"
ONBOOT="yes"

DEVICE=bond0
IPADDR=<ip>
BROADCAST=<ip>
NETWORK=<ip>
GATEWAY=<ip>
NETMASK=<ip>
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED=no

cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 4
        Actor Key: 17
        Partner Key: 11
        Partner Mac Address: 00:24:51:12:63:00

Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 90:e2:ba:0f:76:b4
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 90:e2:ba:0f:76:b5
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 90:e2:ba:0f:76:b6
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth5
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 90:e2:ba:0f:76:b7
Aggregator ID: 1
Slave queue ID: 0

/etc/modprobe.d/bonding.conf

alias bond0 bonding
options bond0 mode=4 miimon=100 updelay=200
#downdelay=200 xmit_hash_policy=layer3+4 lacp_rate=1

Linux:

Linux 3.0.0+ #1 SMP Fri Oct 26 07:55:47 EEST 2012 x86_64 x86_64 x86_64 GNU/Linux

what i've tried:

downdelay=200 xmit_hash_policy=layer3+4 lacp_rate=1

mode 6

Answer 1

Before you start debugging the bonding configuration, have you verified that each network link can work well on its own? If nothing works when you bring eth4 down (ifdown eth4) then there is most likely a problem with your wiring.

Also, you should probably try some of the more simple bonding modes first. Try balance-rr (mode=0) and see if you can see traffic on all interfaces. Also look in /proc/net/bonding/bond0 to verify the current status of your configuration.

Answer 2

You should read this document: Linux Bonding

Answer 3

http://wiki.mikrotik.com/wiki/Manual:Interface/Bonding says:

LACP balances outgoing traffic across the active ports based on hashed protocol header information and accepts incoming traffic from any active port. [...] How hash is calculated depends on transmit-hash-policy parameter.

transmit-hash-policy: layer-2 - Uses XOR of hardware MAC addresses to generate the hash. This algorithm will place all traffic to a particular network peer on the same slave.

So, if you transmitting from bond0 only to a single destination, that is to a single MAC address, all the frames go out from only one of your interfaces. This is an expected behavior. If you'd transfer to some other MAC, chances are that the hash algorithm would select one of 3 other interfaces from your 4 interfaces (chance is 75% to be exact). This ensures the order of delivery of frames.

PS: If your transfer goes through a gateway (a router), this counts as a single-MAC transfer, too. Quite a common scenario.