I have a cluster with 32 machines. The first 25 machines are on the first rack and the rest 7 machines are on the second rack. Each rack has a 1Gbps Ethernet switch. The network communication between the different racks will certainly have a performance penalty (which I don't exactly know)
I used the network performance benchmark tool like 'iperf' to measure the network speed between the machines. There is no problem (all point-to-point connection between 32 machines can exploit the full bandwidth).
However, in my application (which is latency-sensitive with request/respond network communication architecture). The inter-rack network speed is 4~5 times slower than that of the intra-rack network speed.
Is there anything I can do here? Any well-known strategy to apply?