5

I'm trying to utilize maximal bandwidth provided by my 1GiB network card, but it's always limited to 80MiB (real megabytes). What can be the reason? Card description (lshw output):

   description: Ethernet interface
    product: DGE-530T Gigabit Ethernet Adapter (rev 11)
    vendor: D-Link System Inc
    physical id: 0
    bus info: pci@0000:03:00.0
    logical name: eth1
    version: 11
    serial: 00:22:b0:68:70:41
    size: 1GB/s
    capacity: 1GB/s
    width: 32 bits
    clock: 66MHz
    capabilities: pm vpd bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation

The card is placed in following PCI slot:

*-pci:2
     description: PCI bridge
     product: 82801 PCI Bridge
     vendor: Intel Corporation
     physical id: 1e
     bus info: pci@0000:00:1e.0
     version: 92
     width: 32 bits
     clock: 33MHz
     capabilities: pci subtractive_decode bus_master cap_list

The PCI isn't any PCI Express right? It's a legacy PCI slot? So maybe this is the reason?

OS is a linux.

3
  • 8
    It would be useful to describe how you are attaining the benchmark of 80MiB.
    – Dan Carley
    Jun 17, 2009 at 14:34
  • 3
    You probably reached you maximum disk read capacity. Jun 24, 2009 at 9:37
  • 1
    A cheap NIC on a slow bus, raw throughput should reach closer to 120MB/s.
    – Steve-o
    Aug 25, 2011 at 3:09

10 Answers 10

38

80 MB / second is actually pretty good! That's about 640mbps, which is pretty darn close to the gigabit capacity of the NIC. If you take into consideration the TCPIP overhead, and disk speed you're probably at your maximum speed.

9
  • 5
    Correct, rule of thumb when using TCP is 20% overhead.
    – pauska
    Jun 17, 2009 at 14:21
  • 1
    As he talks about 1GiB and 80MiB, I guess it's 80mpbs and not 80MB/s.
    – radius
    Jun 17, 2009 at 14:22
  • 3
    Though he does say "real megabytes" in his original post. Even if it is 80mpbs, he could have slow disk performance as a bottleneck. Jun 17, 2009 at 14:27
  • 2
    I assume "real megabytes" refers to mebibytes, not the fake megabytes that disk manufacturers like. (1024^3, rather than 1000^3) Jun 17, 2009 at 14:57
  • 1
    I saw you corrected 1GiB by 1Gbps, you must not use mebibytes for network but megabytes. Anyway your 80MiB are 671mbps. Now we have to know how you measure this. If it's a speed on ethernet layer it's poor for an Gbps card, if it's a speed at application level it's quite good but you could do a little better but we have to know how you measure it.
    – radius
    Jun 17, 2009 at 15:35
21

Try putting this to your /etc/sysctl.conf

# General 10gigabit/LFP tuning
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_orphans=1048576
net.ipv4.tcp_orphan_retries=2

# Removes some internal buffering
net.ipv4.tcp_low_latency=1

# Time-wait sockets
# Do not turn on unless you know what you are doing!
#net.ipv4.tcp_tw_recycle=1
#net.ipv4.tcp_tw_reuse=1

# If PMTUD ICMP blackhole appears use
# RFC 4821, Packetization Layer Path MTU Discovery
net.ipv4.tcp_mtu_probing=1

# Netfilter's conntrack
# NB! For high-performance concerns you probably don't want to use `--state` rules at all 
#net.ipv4.netfilter.ip_conntrack_max=1048576
#net.nf_conntrack_max=1048576

# SACKs are an optimization to TCP which in normal scenarios improves considerably performance. 
# In Gigabit networks with no traffic competition these have the opposite effect. 
# To improve performance they should be turned off with: 
#net.ipv4.tcp_sack=0 

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout=15
# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time=1800

# Increased backlog (default: 100/1000 depending on kernel)
net.core.netdev_max_backlog=10000
net.core.somaxconn=10000

# Timestamps adds additional 12 bytes to header and uses CPU
# NB! It caused massive problems for me under benchmark load
# with a high count of concurrent connections.
# ( http://redmine.lighttpd.net/wiki/1/Docs:Performance )
#net.ipv4.tcp_timestamps=0

# Portrange for outgoing connections
# (increase the ephemeral port range)
# NB! After that tuning you probably do not want to listen on port >= 1024
net.ipv4.ip_local_port_range=1024 65535

# Fixing 'Too many open files', Second useful on nginx+aio workloads
fs.file-max=16777216
fs.aio-max-nr=65536

# If you are under DDoS you can
kernel.panic=10
# Lower following values
#net.ipv4.tcp_synack_retries=2
#net.ipv4.tcp_syn_retries=2
#net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait=15
#net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait=15
# If you under ping flood
#net.ipv4.icmp_echo_ignore_all=1

Each connection we make requires an ephemeral port, and thus a file descriptor, and by default this is limited to 1024. To avoid the Too many open files problem you’ll need to modify the ulimit for your shell. This can be changed in /etc/security/limits.conf, but requires a logout/login. For now you can just sudo and modify the current shell (su back to your non-priv’ed user after calling ulimit if you don’t want to run as root):

ulimit -n 999999

Another thing you can try that may help increase TCP throughput is to increase the size of the interface queue. To do this, do the following:

ifconfig eth0 txqueuelen 1000

You can play with congestion control:

sysctl net.ipv4.tcp_available_congestion_control
sysctl net.ipv4.tcp_congestion_control=htcp

There is also some low level tuning, e.g. kernel module parameters

# /sbin/modinfo e1000
..snip...
parm:           TxDescriptors:Number of transmit descriptors (array of int)
parm:           TxDescPower:Binary exponential size (2^X) of each transmit descriptor (array of int)
parm:           RxDescriptors:Number of receive descriptors (array of int)
parm:           Speed:Speed setting (array of int)
parm:           Duplex:Duplex setting (array of int)
parm:           AutoNeg:Advertised auto-negotiation setting (array of int)
parm:           FlowControl:Flow Control setting (array of int)
parm:           XsumRX:Disable or enable Receive Checksum offload (array of int)
parm:           TxIntDelay:Transmit Interrupt Delay (array of int)
parm:           TxAbsIntDelay:Transmit Absolute Interrupt Delay (array of int)
parm:           RxIntDelay:Receive Interrupt Delay (array of int)
parm:           RxAbsIntDelay:Receive Absolute Interrupt Delay (array of int)
parm:           InterruptThrottleRate:Interrupt Throttling Rate (array of int)
parm:           SmartPowerDownEnable:Enable PHY smart power down (array of int)
parm:           KumeranLockLoss:Enable Kumeran lock loss workaround (array of int)
parm:           copybreak:Maximum size of packet that is copied to a new buffer on receive 

And even lower level hardware tunings accessible via ethtool(1).

PS. Read kernel the docs, especially Documentation/networking/scaling.txt

PPS. While tuning TCP performance you may want to consult with RFC6349

PPPS. D-Link is not the best network hardware. Try Intel hardware with pci-x or pci-64

8

Your 32-bit, 33Mhz PCI bus can transit a maximum of 1,067 megabits per second (Mbps) or 133.33 megabytes per second (MBps).

Gigabit Ethernet can transit 116 megabytes per second (MBps).

So although you card should be able to fully saturate the line you'll actually only ever get about 90% utilisation because of various overheads.

Either way if you're getting 80 megabytes per second (MBps) then you're not far off and I would be reasonably happy with that for now.

1
  • 4
    not to mention ethernet frame, ip and tcp packet overheads. Jun 17, 2009 at 14:58
2

Gigabit ethernet is just over 1 billion bits per second. With 8/10 encoding this gives you a maximum of around 100MB per second. A 32 bit PCI bus should be able to put 133MB/sec through and you should be able to saturate it (I can demonstrate saturation of a PCI bus with a fibre channel card and get a figure close to the theoretical bandwidth of the bus), so it is unlikely to be the cause of the bottleneck unless there is other bus traffic.

The bottleneck is probably somewhere else unless you have another card using bandwidth on the bus.

3
  • wikipedia says that twisted pair based 1000Base-T doesn't use 8/10. Is it right? Jun 18, 2009 at 2:25
  • Not sure. I was under the impression it did. Maybe I'm thinking of Fibre Channel. Jun 18, 2009 at 10:06
  • 1000Base-T has a line rate of 125M symbols/s, the same as 100MbE. It uses all 4 pairs however, and 5 signal levels (simplified: 2 level signals per direction and a neutral), to produce 1 Gbps. The first 8 bits are encoded to 12 transmission bits to prime the DC-balance algorithm, but otherwise transmissions occur at line rate. 1000Base-SX (and other fiber variants) do use 8b/10b line coding and operate at a line rate of 1.25Gbps.
    – Chris S
    Nov 17, 2011 at 22:32
1

Bottle necks at GigE speeds can come from a number of places.

  • Disk subsystem: It takes at least 3-4 hard drives in a RAID array of some sort to be able to hit GigE speeds. This is true on the sending and receiving end.
  • CPU: GigE can use a lot more CPU than you would think. Given that it's in a 33mhz PCI slot I'm going to go out on a limb here and say that this system is fairly old and may have a slower cpu.
  • TCP/IP overhead: Some bits that are sent over the wire is not the data payload but other overhead bits. This said I have had a system that consistently hit and sustained 115MB/s with a single GigE link.
  • PCI Bus: Is the NIC the only thing on that PCI bus or is it being shared with another device.
  • Other factors: There are too many other factors to mention them all but some of the biggest would be what other disk IO activity is happening. Is it a mix of read/write, lots of small IO requests, etc.
3
  • "at least 3-4 hard drives in a RAID array" -- isn't this only true for disk writes?
    – romandas
    Jul 8, 2009 at 18:09
  • 1
    While writes are more expensive than reads in terms of performance there's still no way a single hard drive is going to keep up with a gigabit network connection. Even a 15k rpm SAS drive isn't capable of sustaining gigabit speeds across the entire surface for reads or writes. Jul 8, 2009 at 18:18
  • 1
    Striping two sata drives should be able to fill a gigabit, even if done in software.
    – Roy
    Oct 20, 2009 at 9:28
0

How sure are you that it is the card that is the bottleneck? It might be that is the best speed it can negotiate with the device on the other end so it is stuck waiting. The other device might be stuck running at 10/100 speeds so 80 would be about right with a bit of overhead.

1
  • 1
    Erm, no. If the other end was running at 10 Mbps, you'd be unable to push more than about 1.2 MB/s (slightly less, in fact) and if it was running 100 Mbps, you'd be looking at a peak somewhere in the 11.9 MB/s range. "Gig ethernet" is not "gigabyte", it's "gigabit" (and it's a "base 10" gig, at that, so 10^9, not 2^30).
    – Vatine
    Nov 15, 2010 at 13:25
0

After my long lasting research I post my conclusions:

  1. Process affinity and NIC irq affinity must be fixed and equal. The irqbalance process (Linux) should be stopped.
  2. The card must be running in its PCIe mode, e.g. x4. If a PCIe slot doesn't support this mode, the card will run in x1 mode resulting in bottleneck.
2
  • First of all, the OP is using PCI not PCIe. Also, x1 is not a bottleneck for a single GigE card.
    – MDMarra
    Sep 30, 2009 at 11:49
  • 2
    @MarkM: Gallus is the OP. (That's why his/her name is in a beige box.) Dec 2, 2009 at 15:01
0

In my experience 80 MiB/s is pretty good. I've not seen much higher speeds no matter what combination of NICs and switches are being used. I remember 100 Mbps showing much the same behaviour. 70-80% utilization was pretty much all you could ask for, though I see gigabit equipment running above 90% in 100 Mbps mode these days.

By comparison, my very first gigabit configuration at home, based on SMC switches and broadcom integrated NICs could barely manage 400 Mbps. Now, years later and using Netgear management switches along with Intel and Marlin NICs I usually find myself in the range of 70-80 MiB/s sustained transfer.

If you need more, consider bonding multiple interfaces.

0

Other thought ... in addition to TCP, perhaps the higher protocol is struggling to keep up. I frequently watch Windows/SMB on my machines and I never have seen >50MB/s however probably its disk limiting here.

One more ... perhaps the network switches are struggling with traffic.

0

your getting very good speed if can check on your switch end

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_configuration_example09186a0080094470.shtml

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .