I have a cluster of machines with two 10Gbps intel network cards on each machine. 64 port Router, with each port of capacity 10Gbps is being used. when i try to measure maximum network performance between machines connected to same router, I see that i get close to 99% bandwidth for a pair of network cards between only 2 specific machines, but i get only around 40-60% bandwidth between rest of pairs of machines.
The driver settings are the same in all machines.
The network test tools i used were iperf and NTttcp.
All machines are identical with respect to CPU, memory, network cards.
I tried swapping the cables of the high network performance bandwidth machines with cables of other machines. That dint seem to effect the results.
Can someone please suggest what might be wrong?