Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am setting up a supercomputing Linux cluster at work. We ran the most recent HPCC benchmarks using OpenMPI and GoToBlas2 but got really bad results. When I ran the benchmarks using one process for every core in the cluster, the results were much worse (more than 100X) than running the benchmark in a single process. This is clearly not the kind of performance we expected. My only thought is that MPI is taking too long to transfer messages among the processes. Does anybody have any ideas of how I can optimize the server setup so that the performance doesn't suck so much?

We are using the Rocks cluster distribution with OpenMPI v1.4.3. Our compute nodes are Dell rack-mount servers with two quad-core Intel Xeon processors each. They are connected by gigabit ethernet cables.

share|improve this question
add comment

1 Answer

When looking at a scientific cluster and performance here are some of the main bottlenecks I see:

  • What kind of networking do you have. Yes you say you have gigabit ethernet but are you using non-blocking switches so that every node on the switch can get full line rates?
  • Are you using a distributed file system or an optimized NAS?
  • Are all your links at full line rate? Again this goes back to the first point but you'd be surprised at what you'll find running iperf occasionally on nodes
  • What is your latency. This can come up as a problem from time to time with gigabit networks if you have network issues and can really put a damper on applications that need to use MPI.
  • What is your settings for your mainline device in network-scripts? Is your MTU set to 9000?

Iperf can normally be found on RHEL systems at

/apps/rhel5/iperf/bin/iperf

To run iperf first setup a server on a node.

/apps/rhel5/iperf/bin/iperf -s

Then from the node you wish to test the link do

/apps/rhel5/iperf/bin/iperf -c <host or IP of server>

If successful you will see output like this on the client:

------------------------------------------------------------
Client connecting to <host or IP of server>, TCP port 4200
TCP window size:   256 KByte (default)
------------------------------------------------------------
[  3] local 123.11.123.12 port 4400 connected with 123.456.789.12 port 4200
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.1 GBytes  1.01 Gbits/sec

If you don't have it installed it can easily be retrieved on many platforms from repositories and is freely available to download and compile from source if it isn't. Run this on each node to see if there is a problem with the actual ethernet wire. After that run it on all the nodes to see if it bogs down the switch.

share|improve this answer
    
Wilshire, thanks for your answer. I didn't set up the networking (it was setup before I started) so I'm not really sure about the specifics. I'm pretty sure it is a latency issue but I am not certain how to fix that. I'll talk to my boss about the networking and see if he can answer your questions. –  Zhehao Mao Jun 20 '11 at 18:45
    
As for the results of the HPCC benchmark. I've put them here. I ran the benchmarks using 128 processes on 16 8-core compute nodes. –  Zhehao Mao Jun 20 '11 at 18:48
    
Just browsing the HPL output stuff looked really off. You should be getting at least over 700 Gflops on 128 processors. –  Wilshire Jun 20 '11 at 19:10
    
Yeah, I know, that's what my boss said. Instead the performance goes in the opposite direction. With just one processor, it get 8 Gflops, but with 128 processors it gets only 30 Mflops. –  Zhehao Mao Jun 20 '11 at 19:18
    
Yeah, if this is the case there might be something wrong with the network. Run iperf on all the nodes to check for bad cabling. –  Wilshire Jun 20 '11 at 19:22
show 6 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.