On our 24 core bl685 (4sockets x 6core), we find that NUMA nodes 0 and 1 are pretty busy (unfortunately resulting in elevated cpu ready times on the VMS), whilst NUMA nodes 2 and 3 are almost unused.

I thought this just maybe a ESX4 U1 issue, so I had a colleague with a 32 core (dl785) farm investigate, and it seems that his last 3 or 4 NUMA nodes are also not really being utilised.

ESX seems to have a weakness when it comes to balancing lightly loaded NUMA boxes, Im going to enabled node interleaving in the BIOS and see if the scheduler balancers across all 24 cores, instead of just 12!...

For those of you with large core counts, I would suggest you fire up you viclient, and check Physical CPU useage (or esxtop), I would be interested to hear what your results are. Please note, that its only the lightly loaded (eg less than 30% cpu load on the esx host) that seems to have the biggest issue with load imbalance.


PS ive logged a SR with vmware to assist, also the other "problem" could be that we have 128gb of ram in each host, and therefore the scheduler sees no good reason why it shouldnt try and cram all vms's into the first two NUMA nodes, as we only have around 50gb of ram worth of vms on each host...

  • @babyg_wc: welcome to Serverfault! While your post is interesting, it's not clear what your question is. SF is a Q&A site and we normally close all non-question posts where there cannot be specific answers (see serverfault.com/faq ). Maybe you change (edit) your post in order to make clear what you're asking. Thank you!
    – splattne
    Feb 23 '10 at 11:49
  • Original source: communities.vmware.com/message/1481534. You may be interested in this question on meta: meta.stackexchange.com/questions/43096/…
    – Helvick
    Mar 28 '10 at 17:21
  • I am having the same issue on BL685 G7 blades with 48 cores. All the VMs are getting dumped into Node 0 and causing unacceptable CPU ready times. Only 6 of the 48 cores are really showing any activity (around 30%). I have an SR opened with VMware as well and waiting to troubleshoot.
    – user73094
    Mar 4 '11 at 14:08
  • Update: VMware support is generating a PR and this is going to their engineering team. The problem was easily reproduced. I am collecting performance and logs for them now.
    – user73094
    Mar 4 '11 at 15:38
  • I know this is an old post but in case someone else comes across this there's now a VMwareKB article about it; kb.vmware.com/kb/1026063 Regards, Ed
    – user75321
    Mar 21 '11 at 23:30

I don't have experience with NUMA, but we do run several "high core count" ESX 3.5 servers.

We find that core 0 is usually used by the console, so do you have anything running on the console host that could explain this?

For example, we run esxtop in batch mode to record very low level stats, and these stats are gzipped up nightly, and this is all on CPU/core 0.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy