I'm running a vSphere private cloud configuration at a managed hosting vendor. The physical hosts have dual 14-core CPUs and 128 GB of RAM each.

An application that we run can multi-thread expensive computational tasks, and I have requested the vendor to create three VMs with 20 vCPU each, and 32 GB of RAM. Note that the vCPU to physical core ratio will remain extremely low, not much bigger than 1, and total RAM will be undersubscribed by a healthy amount.

Engineers at the vendor say that a 20-vCPU VM will negatively impact performance because it spans more than one physical (14-core) CPU socket, even though there is a total of 28 physical cores available on each host. This makes no sense to me, but I don't know enough about this and generally rely on the vendor recommendations. Are they correct about this warning?

  • 1
    Yes, you'll take a performance hit. Read up on NUMA to understand the underlying physical design of multiple CPU servers, and VMware's Virtual NUMA to understand how such a VM will operate in this environment. – Michael Hampton Jul 18 '16 at 20:28

This is a generally bad idea unless you can really justify the number of CPUs.

Please place the onus back on the provider. What do they recommend you do, based on the workload?

NUMA is less of an issue. But in practice, very few VMs in my environments have ever needed that number of CPUs. There were certainly more options and places to optimize before getting to that point.

I've been on both sides of this type of infrastructure. It sucks that this is what you need for your application... but it's really the provider's duty to make it work (or set maximums on parameters).

You're essentially forcing a one VM-per-host situation because of the size of the proposed virtual machine. That's the main reason the engineers are pushing back. The host becomes considerably less useful if it can only accommodate one of these virtual machines.

  • I've heard that giving a VM a vCPU count that approaches or equals the total number of physical cores on a host is generally a bad idea, but I don't know of any rule of thumb that says not to give VMs a vCPU count that exceeds the number of cores per socket on the host. Can you explain more or provide a document reference? OP says they have an app that does expensive computation. Indeed, VMs with lots of vCPUs are common in scientific computing, so I'm very curious about this answer. – JasonAzze Jul 19 '16 at 13:00
  • Well, there aren't many technical limitations, and VMware is NUMA aware... However, this is more of an issue that may be an indicator of bad design. E.g. if the VMs require a certain core count, should the workload be virtualized? What are the implications of vmotion across the cluster when your VMs are mis-sized? Should the OP's provider be looking at quad-socket host servers, etc. – ewwhite Jul 19 '16 at 13:07
  • @ewwhite The application runs compute-intensive tasks that take minutes or (in extreme cases) hours on a single thread, and currently the only way to provide better service is through multi-threading. Agreed that in the longer term a fundamental redesign that can take advantage of MPP frameworks would be appropriate. But in the short term it's not a viable business option - hence the desire to scale CPU count up so that long client tasks complete faster via multi-threading. – O.Sheyner Jul 19 '16 at 16:23
  • This is going to be the vendor's responsibility. As someone who's been on both sides of this infrastructure, it sucks that this is what you need... but it's the provider's duty to make it work. What's basically happening is consolidation is impacted because you're essentially creating a one VM-per-host situation. That's the main reason they're pushing back. – ewwhite Jul 19 '16 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.