1

I have a large job and I'm experimenting with the topology to see what gives better results (playing with ntasks, ncores-per-cpu, nodes, etc.) I'm using slurm as my work queue manager.

I have two nodes (CentOS 7, managed with Bright 7), each have 2xIntel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz. So, two processors with 12 cores, all hyper threaded, so I appear to have 48 cores per node. (See below for the processor topology lstopo gives)

My question: should I disable hyperthreading in the bios, or try to disable it in SLURM? Or just treat my system like I have double the cores? Specifying --thread-per-core=1 doesn't seem to have an effect.

My job is a big environmental model, lots of I/O, lots of matrix calculations, etc, and takes on the order of days right now to run.

I've been reading the SLURM FAQ, but am still confused how to proceed.

Processor topology showing NUMA spaces

closed as unclear what you're asking by Iain, Ward, MadHatter, mdpc, HopelessN00b Mar 10 '16 at 7:05

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    What on earth are you talking about? AMD processors do not do hyperthreading. – Michael Hampton Mar 7 '16 at 4:55
  • ACK, I'm dumb. I was running lscpu on the headnode, and not on the Intel node. I edited the question to reflect this. (shouldn't have wrote this question at 1 am :( ) – Matt Mar 7 '16 at 16:17
  • 1
    Well then. It really depends entirely on your workload. Put together a job that takes ~ 1 hour or so on the current configuration, and then run it without hyperthreading and see how long it takes. The answer should then be obvious. As a general rule, though, hyperthreading wins for most things. – Michael Hampton Mar 7 '16 at 16:28
1

Basically, if you feel that your operations would benefit from the freer L2/L3 cache that is possible when HT is disabled, go ahead and disable HT from the BIOS.

If however there is a large number of small operations that can be run in parallel, you are definitely benefiting from leaving HT enabled.

VMWare ESXi and Hyperthreading

http://lifehacker.com/how-hyper-threading-really-works-and-when-its-actuall-1394216262

1

Slurm allocates by cores, SLURM was designed for portability not performance. So the way it allocates is "Note that even on systems with hyperthreading enabled, the resources will generally be allocated to jobs at the level of a core (see NOTE below). Two different jobs will not share a core.". It does know what hyperthreading is and it can be enabled but its specifically looking at cores per socket. It is open source and I'm sure a hyperthread optimization is in order.

  • I read that, but I was still having trouble making it target two nodes if it felt that one node wasn't full. Maybe htop isn't the best tool to measure how it's running. Reading this, and the previous answer, I think that I really just need to keep experimenting with different combinations. I'm getting the impression that hyper-threading is helping me out though, but still am not sure. – Matt Mar 7 '16 at 19:12
  • It might be, from slurms point of view its ignoring it, and if you are using slurm, performance isn't the goal anyway. – Jim B Mar 7 '16 at 19:48
  • What do you mean by that? I was under the impression that slurm shouldn't impact performance if set up properly. Right now my stuff is taking way too long so performance is key. I'm trying to run a model (government written Fortran model) whose setup scripts require a workqueue manager, and slurm is the one our cluster came with.... Would a different queue make a big difference? – Matt Mar 8 '16 at 2:20
  • 1
    it could, Univac, is my recommendation when performance is required, it has phi support and gpu support, and CPU scavenging. requests for product recommendations are off topic however. – Jim B Mar 8 '16 at 8:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.