Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm working with the bonding driver on Linux (Ubuntu 12.04.2 LTS).

I have 2 PCs with 2 NICs (gigabit ethernet - Intel 82574L) in each one (in total, 4 NICs). I want to improve my TCP throughput over a gigabit network, using bonding, but my problem is that I can't achieve gigabit speeds using bonding.

My gigabit interface can reach 370 - 375 Mbps using a single gigabit interface (without using bonding). When I turn on my bonding, using mode 0 (balance-rr), for example, I can reach 760 - 772 Mbps (using iperf).

When I want to transfer data from one PC to another (using bonding mode balance-rr), I just can reach 472 Mbps (59 MB/s). To tranfer data, I use scp.

My questions are:

  1. What are the limitations of gigabit ethernet over Linux?
  2. Why can't I achieve gigabit speeds?
  3. Can anyone help me to change the default settings on Ubuntu to achieve gigabit speeds?

Can you advise on how to tune Ubuntu for achieve gigabit speeds?

My sysctl.conf:

# cat sysctl.conf | grep -v ^# | tr -s '\n' 
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

# ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

# ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX:        4096
RX Mini:    0
RX Jumbo:    0
TX:        4096
Current hardware settings:
RX:        256
RX Mini:    0
RX Jumbo:    0
TX:        256

# ethtool -g eth2
Ring parameters for eth2:
Pre-set maximums:
RX:        4096
RX Mini:    0
RX Jumbo:    0
TX:        4096
Current hardware settings:
RX:        256
RX Mini:    0
RX Jumbo:    0
TX:        256
share|improve this question
2  
You could potentially be maxing out on your disks. How are your disks configured? –  Nathan C Jun 7 '13 at 16:09
    
Or possibly the CPU if you are using an encrypted transfer protocol. You should test with iperf. –  Zoredache Jun 7 '13 at 18:32
    
+1 - Even on fast machines there is a substantial penalty for scp vs just using netcat or even FTP. –  rnxrx Jun 8 '13 at 2:36
add comment

1 Answer

A single stream can only ever be received on a single link. So while your links may be bonded and the packets split in a round robin fashion on the way out, they are still received by just a single link on the other side.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.