I've connected two hosts via a 40Gbe fibre link and would like to determine a baseline for data transfer between them. I'm seeing relatively low speeds (~620MB/s). It seems that the signalling rate is lower and may in fact be 5Gb/s (which would agree with what I'm seeing), but I know very little about fibre (and am on shaky ground).
My setup is as follows:
An extract of the output of ethtool on the 40Gbe interfaces, on each node, is
Speed: 40000Mb/s
Duplex: Full
Port: FIBRE
Running a UDP receiver on one host (10.0.0.2)
nc -vv -u -l 2222 > /dev/null
and a sender on the other
time dd if=/dev/zero count=10000 bs=1500k | nc -u 10.0.0.2 2222
results in speeds around the 650MB/s mark. Different block sizes make small differences.
If someone could point out if my approach or thinking is wrong, I'd be most grateful (I'm very green in this area and wiki pages and manufacturer docs have been a bit opaque for me).