1

My guest OS shows about 5 percent CPU usage, but my host machine shows the QEMU process taking about 25 percent.

I'm trying to understand where this difference comes from, as I am using KVM acceleration and the CPU should have near-native performance.

I think the disk or network access may contribute to this difference, as these layers are invisible to the guest.

But is there a way I can determine exactly what is the main cause of this 20 percent overhead?

1 Answer 1

1

CPU utilization ratios rarely tell the entire story. Some tools count fractions of 1 CPU, some fractions of all system CPUs. The total does not explain what the CPU/memory load is doing, it could be doing useful work, it could be system overhead.

A better test would quantify the amount of work done by some application, while pushing the system hard. And profile the system while this happens. Stress test microbenchmarks create load on demand, even if these "bogus" operations are synthetic and a little artificial.

For example, on Linux perf record -a -F 999 -- stress-ng --metrics --cpu 1 --timeout 1m will create one minute of CPU load while profiling what is on any CPU at millisecond precision. "bogo ops" quantify work done.

Repeat this test multiple times, in a VM, and on a bare metal OS install. Remember scientific method, keep as many variables the same as possible. Identical hardware, same OS distro and patch level, same workload parameters, same profiling parameters.

Review what is on CPU with perf report Bucketing these into kernel, drivers, and the application requires installing debug symbols, and some research into what various functions are doing. Hypervisor overhead will not appear in the guest, but can be inferred by any throughput difference with the bare metal results.

Synthetic testing will be very different from real workloads. My stress-ng example does zero I/O, which never happens in practice. Do similar profiling and analysis, but the workload is what the host is supposed to be doing. Hint: when profiling the entire system, but not needing to start another program, can have sleep run a timer: perf record -a -- sleep 60

2
  • I really appreciate the effort you took for your answer, but it feels like its answering a totally different question. I am not doing any benchmarks at all. I just noticed that there is a large difference between what the guest is reporting in top, and what the host is reporting. So I just wondered what could be possible causes for this difference. I expect it is very normal what Im seeing, I was just curious.
    – Maestro
    May 12, 2023 at 19:04
  • Possibly you are mistaking fractions of 1 CPU metrics with fractions of total CPU metrics. The rest of my answer is the question you asked, how to quantify over: really run it on bare metal vs in a VM, and that being the only difference, and profile to see where resources are being spent. Note I conclude that profiling bare metal vs VM can be done to real workloads as well. May 13, 2023 at 15:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .