3

I've connected two hosts via a 40Gbe fibre link and would like to determine a baseline for data transfer between them. I'm seeing relatively low speeds (~620MB/s). It seems that the signalling rate is lower and may in fact be 5Gb/s (which would agree with what I'm seeing), but I know very little about fibre (and am on shaky ground).

My setup is as follows:

An extract of the output of ethtool on the 40Gbe interfaces, on each node, is

Speed: 40000Mb/s
Duplex: Full
Port: FIBRE

Running a UDP receiver on one host (10.0.0.2)

nc -vv -u -l 2222 > /dev/null

and a sender on the other

time dd if=/dev/zero count=10000 bs=1500k | nc -u 10.0.0.2 2222

results in speeds around the 650MB/s mark. Different block sizes make small differences.

If someone could point out if my approach or thinking is wrong, I'd be most grateful (I'm very green in this area and wiki pages and manufacturer docs have been a bit opaque for me).

2 Answers 2

2

You would be better off using a tool like iperf instead of dd | nc.

That way you can test both TCP and UDP and get a proper throughput report, as well as controlling the rate, packet size, etc.

People regularly use this to measure 10Gbe and 40Gbe performance.

2
  • Thanks. iperf got me up to ~1350MB (reported 10.8Gb/s). Dan's suggestion got me to similar numbers but I'm still not getting near the maximum speed (or am I)?
    – 0_0
    Apr 25, 2016 at 15:25
  • Changing MTU made quite a difference.
    – 0_0
    Jul 4, 2016 at 13:09
2

Watch top while running your test. It is possible you are hitting CPU limits on the nc command. We've seen nc become the limit to throughput by being CPU bound. Because we observe much higher throughput on kernel space things like DRBD replication, my guess is that nc experiences the overhead of many system calls.

If this is the case, you can run multiple concurrent dd | nc to take advantage of multiple CPU cores and push your test further.

2
  • I checked htop and dstat -cn on both machines and found that CPU utilization was below 10% for a single dd | nc. Interestingly, when I attempt what you suggested by running [receiver]let n++; nc -vv -u -l 222$n > /dev/null & and [sender]let n++; time dd if=/dev/zero count=100000 bs=1500k | nc -u 10.0.0.2 222$n & a few times. I found that the throughput increased and plateaued at ~1470MB/s with 30% CPU utilisation, for three dd threads, after which additional runs contributed nothing.
    – 0_0
    Apr 25, 2016 at 15:12
  • Interesting that you were able to push it a bit further with some concurrency. Have you tried varying the block size to see how it affects throughput? Try powers of two, 4096, 8192, 16384, 32kB, ... 1MB and see how it affects throughput on a single dd. Note: dd was using decimal version of "kilobyte" (1000 instead of 1024), for binary (1024-based) use "kB" and "MB". See iceflatline.com/2010/08/… Apr 26, 2016 at 3:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .