5

I have two Linux machines, each equipped with a Solarflare SFN5122F 10GbE NIC. The two NICs are connected together with an SFP+ Direct Attach cable.

I am using netperf to measure TCP throughput between the two machines. On one box, I run:

netserver

and on the other:

netperf -t TCP_STREAM -H 192.168.x.x -- -m 32768

I get:

MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.x.x (192.168.x.x) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  32768    10.02    1321.34   

The measured throughput is 1.3Gb/s. This is 7.5x below the theoretical maximum, and only 30% faster than 1GbE.

What steps can I take to troubleshoot this?

7
  • 1
    Did you enable jumbo frames?
    – gekkz
    Apr 10, 2012 at 20:08
  • Do you get low throughput on UDP tests as well? Apr 10, 2012 at 20:10
  • @gekkz: Changing MTU to 9000 on both ends made no perceptible difference.
    – NPE
    Apr 10, 2012 at 20:12
  • @ShaneMadden: UDP throughput is about the same.
    – NPE
    Apr 10, 2012 at 20:12
  • 1
    What is the hardware of the boxes involved? You cannot achieve 10GbE line rate with anything less than modern Westmere/Nehalem/Sandy Bridge boxes.
    – pfo
    Apr 10, 2012 at 20:55

1 Answer 1

6

few things:

  • did you try adjusting mtu to make use of jumbo-frames?
  • are you absolutely sure that the link between the two servers does not have any packet losses?
  • does ethtool show you any errors on the interfaces on both ends?
  • what does top/atop say during the prolonged test - do you see any of the cores fully occupied by iowait?

you'll [most probably] not achieve full 10gbit on a single tcp session but you can do some additional tuning to get closer to it by tweaking tcp handling and communication with the network card itself - take a look here or here .

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .