I recently had a couple of 10Gbit Ethernet cards installed on a few machines connected to a LAN of about 80 commodity nodes running a distributed file system (Lustre). The 10Gbit card achieves good performance on file operations and is functioning as it should.
However, I wrote a custom client app in C that asynchronously sends large blocks of data to multiple nodes in the network. The machine the client app runs on has a 10GB Ethernet card, and all the destination nodes have a 1GB Ethernet card - so I should be able to get a theoretical max send transfer rate of 10 gbits.
If I run the client app on a machine with a 1Gbit card, it easily maxes out the card for sustained lengths of time. But strangely, if I run the same app on a machine with a 10Gbit card it gets horrible performance (around 20-30 mbits a second).
The program is written in C using normal TCP sockets. Is there some special setting required for 10Gbit? Because it's very odd that it gets maximum performance on a 1Gbit card, but horrible performance on a 10Gbit card. Again, the problem is not the 10Gbit card itself, since the distributed file system (Lustre) gets good performance with the 10Gbit card.