To give you an idea what kind of answers I would appreciate here are some sub-questions (with links):
- How to you tune the parameters (
Q, memory-alignment, etc) for the
HPL.datfile (without spending too much time trying each possible permutation - esp with large problem sizes N)?
- Are there any Top500 submission rules to be aware of? What is allowed, what isn't?
- Which MPI product, which version? Does it make a difference?
- Any special host order in your MPI machine file?
- Do you use CPU pinning?
- How to you configure your interconnect? Which interconnect?
- Which BLAS package do you use for which CPU model? (Intel MKL, AMD ACML, GotoBLAS2, etc.)
- How do you prepare for the big run (on all nodes)? Start with small runs on a subset of nodes and then scale up? Is it really necessary to run LINPACK with a big run on all of the nodes (or is extrapolation allowed)?
- How do you optimize for the latest Intel/AMD CPUs? Hyperthreading? NUMA?
- Is it worth it to recompile the software stack or do you use precompiled binaries? Which settings? Which compiler optimizations, which compiler? (What about profile-based compilation?)
- How to get the best result given only a limited amount of time to do the benchmark run? (You can block a huge cluster forever)
- How do you prepare the individual nodes (stopping system daemons, freeing memory, etc)?
- How do you deal with hardware faults (ruining a huge run)?
- Are there any must-read documents or websites about this topic? E.g. I would love to hear about some background stories of some of the current Top500 systems and how they did their LINPACK benchmark.
I deliberately don't want to mention concrete hardware details or discuss hardware recommendations because I don't want to limit the answers. However, feel free to mention hints e.g. for specific CPU models.