2

I know this question has been asked somewhat similarly before but my question is more specific and the existing ones are old. So things might have changed a lot (thinking about vector instructions for example).

Simply said I have a python module I basically always need to use and my python code runs a lot slower (double runtimes) on VMs (type 1 and 2). The module itself is mostly a wrapper/API on C library but not excursively.

I'm trying to figure out if python itself is affected or just the module. So is it known that python suffers a lot when running in a VM?

Improve this question

3 Answers 3

Reset to default
1

There's no way from your question to know whether you're comparing apples to apples.

In a properly configured and reasonably loaded virtualization environment, I expect most loads to run at most a few percent slower than on bare metal. If your code is massively scalable and can utilize all available hardware resources, I expect it to perform significantly worse in a virtualized environment, especially if the resources are scarce. If your code is dependent on specific accelerator hardware, the impact of virtualization is implementation specific.

Improve this answer
1
  • Code is single-threaded but I can't really know what the 3rd party module actually does albeit the affected methods I isolated almost certainly can't use any special modern instructions. Hardware resource is also not scare especially on type 2 VM which is vmware workstation on my machine. And it's consistently worse and in this case also the same hardware. And I see the same effect on an esx based VM running on "central infrastructure" which makes me assume it's setup correctly. I have to assume it's really the module causing the problem?
    – beginner_
    Jan 23, 2020 at 11:43
0

You haven't proven anything, you have tons of variables you have not isolated. Python version, operating system version, hardware resources, VM resource quotas, CPU model, and on and on.

Profile what functions in your code specifically are taking the time. Visualize it with flame graphs. Python based stack sampling implementation would be nice, but I don't know how well that captures C functions. You may want to try operating system based profilers (eBPF, xperf) in that case, which have better visibility into the C library and the OS.

Study in detail what functions are slow. Get an idea of what it does, get the source code if possible. Count system calls to measure what it is requesting of the operating system.

Find the limiting resource: CPU, memory, disk, network. Measure those resources at the host level via performance monitoring tools.

Contrast the results on different environments, bare metal, different types of VMs, different hardware. Do not overcommit resources on your VM hosts, not CPU and definitely not RAM. That is unfair when compared to a bare metal dedicated host.

Really, this is a general performance investigation to discover what the limiting factor is in your environment. Using Python just may change the code runtime to profile and optimize.

Improve this answer
2
  • I'm simply a user of a module not a profiling expert. Python version and module version obviously is identical and as mentioned the issues happens on a VM on my laptop and on the native OS so hardware is identical. Code is all in memory. no network or hard disk access needed which is why the large differences of >50% are confusing and should IMHO not exist at all.
    – beginner_
    Jan 24, 2020 at 8:03
  • Then learn profiling or application performance monitoring tools. A thing running 2x slower on a virtual machine is a problem description. A profiled stack showing what function is 20x slower is a diagnosis.
    – John Mahowald
    Jan 24, 2020 at 13:31
0

The chances are that the code in question does a LOT of context switching (and/or the hypervisor is exacerbating the context switching by moving the virtual core around physical cores). Context switching can be up to about 100x more expensive under a hypervisor than on bare metal. Cache misses can also be a large factor, due to the hypervisor scheduling virtual cores around different physical cores.

To alleviate some of those overheads, pin the virtual cores to physical cores, expose the underlying CPU topology to the VM (sockets/cores/threads), keep the VM entirely within a single NUMA node, and don't give the guest the full set of CPU cores/threads you have on the host if your code is cache/memory latency sensitve.

Performance under a hypervisor for various workloads can be dramatically worse than on bare metal. The numbers there are from years ago but I re-test these things reasonably regularly and things haven't changed a great deal in the past decade.

Improve this answer
2
  • Thanks for this Answer. It's for me the first one that makes some sense with what I'm actually observing. To do what you suggest, I would need full access the the hypervisor right?
    – beginner_
    May 4, 2020 at 9:55
  • Correct, you would need access to the host server, you cannot perform these optimisations from inside the guest VM. What might make a little bit of difference is if you use taskset to pin your running process to a single core, assuming it is single-threaded. Or pin each thread statically each to a different core, statically. That won't help you with the overhead induced by the hypervisor switching your virtual cores around, but at least it will remove one layer of switching.
    – Gordan Bobić
    May 4, 2020 at 10:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .