Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two Linux machines, each equipped with a Solarflare SFN5122F 10GbE NIC. The two NICs are connected together with an SFP+ Direct Attach cable.

I am using netperf to measure TCP throughput between the two machines. On one box, I run:

netserver

and on the other:

netperf -t TCP_STREAM -H 192.168.x.x -- -m 32768

I get:

MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.x.x (192.168.x.x) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  32768    10.02    1321.34   

The measured throughput is 1.3Gb/s. This is 7.5x below the theoretical maximum, and only 30% faster than 1GbE.

What steps can I take to troubleshoot this?

share|improve this question
1  
Did you enable jumbo frames? –  gekkz Apr 10 '12 at 20:08
    
Do you get low throughput on UDP tests as well? –  Shane Madden Apr 10 '12 at 20:10
    
@gekkz: Changing MTU to 9000 on both ends made no perceptible difference. –  NPE Apr 10 '12 at 20:12
    
@ShaneMadden: UDP throughput is about the same. –  NPE Apr 10 '12 at 20:12
1  
What is the hardware of the boxes involved? You cannot achieve 10GbE line rate with anything less than modern Westmere/Nehalem/Sandy Bridge boxes. –  pfo Apr 10 '12 at 20:55

1 Answer 1

few things:

  • did you try adjusting mtu to make use of jumbo-frames?
  • are you absolutely sure that the link between the two servers does not have any packet losses?
  • does ethtool show you any errors on the interfaces on both ends?
  • what does top/atop say during the prolonged test - do you see any of the cores fully occupied by iowait?

you'll [most probably] not achieve full 10gbit on a single tcp session but you can do some additional tuning to get closer to it by tweaking tcp handling and communication with the network card itself - take a look here or here .

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.