What is the the purpose of using paravirtualization if there is a hardware assisted virtualization?

Question

I am doing a research about QEMU/KVM and Firecracker/KVM. As far as I understood, both Firecracker and QEMU communicates with KVM to eventually benefit hardware assisted virtualization by changing CPU mode to guest to host and vice versa.

1. In guest mode of the CPU, guest can directly execute even its privileged instructions so why do we also need paravirtualization?

2. In Firecracker, only 5 devices emulated, such as

• virtio-net,
• virtio-block,
• virtio-vsock and so on.

Even in this minimalist design, we have to put paravirtualization drivers. Can't we just depend on hardware assisted virtualization?

Answer 1

Consider just the case of networking.

To actually be useful in most cases, a VM needs to be able to communicate over the network. For that, the guest obviously has to see some sort of network interface. But VT-x and AMD-V and ARM VHE and pretty much all other hardware virtualization implementations don’t provide NICs, they just give you a way to securely segregate and partition out CPU resources. So hardware virtualization does nothing to give you a network interface.

Now, you could pass through a physical network interface from the host system, but that has a number of issues:

• It requires special handling at boot time for the host OS to ensure that it doesn’t actually bind a driver to that interface.
• It requires special support in the hardware to be secure (you need an IOMMU, and the NIC you want to pass through has to support running behind an IOMMU).
• It requires actually having a usable network interface for each VM. This obviously means that anybody on a laptop would be out of luck, but it also means that most large scale VM hosting providers would be out of luck as well (they may be running many dozens of VMs on the same host).
• It makes live migration functionally impossible without RDMA hardware and a lot of additional complexity (just like any other direct device pass-through does).
• In the case of VMs needing to talk to each other, it inherently introduces an external point of failure (because they have to send traffic through a network switch outside of the system).

So you obviously need to emulate a network interface somehow. The obvious choice would be to just pick a commonly used physical NIC and emulate that. But that has it’s own set of issues:

• A lot of what a physical NIC does is rather computationally expensive to emulate. It’s only efficient in a physical implementation because it’s using physical logic and ASICs.
• A majority of the same stuff that’s expensive to emulate isn’t even needed for a VM, but you can’t avoid emulating it because the drivers will expect it to work.
• A lot of additional complexity is needed in the guest drivers to support this stuff that isn’t even giving any real benefit. For perspective, the Intel e1000 (a commonly emulated physical NIC) driver for Linux is about 17k lines of code, while the virtio-net driver is only 4.8k (7.3k if you include the virtio-pci components that it is probably using on x86 systems).

virtio-net solves those issues, it only covers the things that are actually needed to move network packets between the guest OS and the host networking layer, and nothing more. And solving those issues provides a huge performance improvement. I have not tested recently, but the last time I compared it using QEMU, virtio-net provided more than twice the effective bandwidth of an emulated e1000 card, and roughly 1/10th of the latency, all with lower CPU usage on the host side.

The same logic applies for most other devices. Some stuff can be emulated relatively inexpensively or is not performance critical and thus doesn’t need to be efficient (this is why there is no VirtIO watchdog timer for example, it’s not performance critical, and it’s trivial to emulate in most cases), but for most things that don’t fit those criteria there is a paravirt option because the performance difference is huge and the reduced complexity tends to make things more reliable.

And sometimes paravirtualization lets you do things you couldn’t really do with ‘regular’ hardware. VirtIOFS and the VirtIO transport for 9P2000 are prime examples of this, they have no hardware analogues, but provide a reasonably efficient way to share files between the host and guest without needing to emulate a network or a block device.

Answer 2

Short answer: when augmented with guest-side drivers, modern hardware virtualization uses some form of paravirtualization for the most performance-intensive operations. Hence modern hypervisors gathers both from dedicated hardware support and paravirtualization.

Long version: the original X86 arch was quite difficult to correctly virtualize and use with unmodified OS. One solution was to dinamically translate the problematic code fragments, recompiling them on the fly. This enabled unmodified guest to run, but the downsides were high overhead and complexity of the translator itself. Moreover, each guest kernel type needed special treatment.

For this reason, projects started to modify the Linux kernel to avoid the difficult cases - ie: when an instruction/call Is not easily virtualized, lets call an hypercall from the underlying hypervisor. An hypercall is the foundation of paravirtualization, and you can consider it the equivalent of userspace syscall. In other words, the underlying Linux kernel was the host OS where another Linux kernel would run as a guest "userspace" application.

This approach minimize virtualization overhead, but it needs a modified guest kernel to work. Your standard Windows installation will not work. So full hardware virtualization was introduced, where additional privilege rings were added to the microprocessor. This permitted the host OS to run in a specifically privileged level (ie: -1, hypervisor space), with the guest OS running unchanged in ring 0 (ie: 0, kernel space). However virtual device emulation remain problematic and/or with high overhead, so custom guest drives where created. These drives reintroduced targeted para-virtualization for the most performance sensitive devices - disk and network, specifically. That bring us to the current situation, where hardware-assisted hypervisors are augmented with targeted paravirtualized drives.

Answer 3

I've done some research as well, and I think it might help you understand the thing, but it might not address it directly (maybe). In Addition to shodanshok answer, i may blow up his "extended" answer maybe a bit of a huge footprint

Paravirtualization (may) remains relevant for several reasons:

1. Performance Optimization:

Even though privileged instructions can be executed directly in guest mode, paravirtualization can enhance performance by optimizing communication between the guest OS and the hypervisor. Paravirtualization allows for the use of specialized interfaces and protocols designed for efficient collaboration between the guest and the hypervisor.

2. Flexibility and Portability:

Paravirtualization enables virtualizing guest operating systems that may not be optimized for direct execution in a virtualized environment. By providing specific interfaces, guest systems can be more easily ported across various virtualization platforms.

3. Security:

Paravirtualization can offer security benefits by providing a well-defined interface for communication between the guest and the hypervisor, potentially reducing attack vectors.

4. Regarding Firecracker and similar minimalist virtualization solutions:

Even though they operate with a limited set of emulated devices, the use of paravirtualization allows for more efficient and flexible communication between the guest and the hypervisor. Even in minimalist designs, paravirtualization drivers can improve overall performance and optimize the interaction between guest and hypervisor. Therefore, paravirtualization remains beneficial even in such environmen

Wikipedia Say about it:

In computing, paravirtualization or para-virtualization is a virtualization technique that presents a software interface to the virtual machines which is similar, yet not identical, to the underlying hardware–software interface.

The intent of the modified interface is to reduce the portion of the guest's execution time spent performing operations which are substantially more difficult to run in a virtual environment compared to a non-virtualized environment. The paravirtualization provides specially defined 'hooks' to allow the guest(s) and host to request and acknowledge these tasks, which would otherwise be executed in the virtual domain (where execution performance is worse). A successful paravirtualized platform may allow the virtual machine monitor (VMM) to be simpler (by relocating execution of critical tasks from the virtual domain to the host domain), and/or reduce the overall performance degradation of machine execution inside the virtual guest.

Paravirtualization requires the guest operating system to be explicitly ported for the para-API – a conventional OS distribution that is not paravirtualization-aware cannot be run on top of a paravirtualizing VMM. However, even in cases where the operating system cannot be modified, components may be available that enable many of the significant performance advantages of paravirtualization. For example, the Xen Windows GPLPV project provides a kit of paravirtualization-aware device drivers, that are intended to be installed into a Microsoft Windows virtual guest running on the Xen hypervisor.[1] Such applications tend to be accessible through the paravirtual machine interface environment. This ensures run-mode compatibility across multiple encryption algorithm models, allowing seamless integration within the paravirtual framework.

I think, even though we can refer to the Wikipedia article, there might be reasons why you wouldn't want to expose the hardware directly to the guest, especially if you're selling rented virtual machines.

Additionally and normally, we want standards and standardized drivers as also interfaces.
We don't want to usually deal with drivers like we did, while trying to install Windows 9x/nt/VIST/XP. But more or less also 7/10/11 on standard hardware (they mostly already have, sometimes the VirtIO support already included). (Specially before Windows 7 period times).

That's why we need to learn more about history:

The History Section of the Wikipedia Article says:

The Parallels Workstation operating system calls its equivalent a "hypercall". All are the same thing: a system call to the hypervisor below. Such calls require support in the "guest" operating system, which has to have hypervisor-specific code to make such calls.

The term "paravirtualization" was first used in the research literature in association with the Denali Virtual Machine Manager.[4] The term is also used to describe the Xen, L4, TRANGO, VMware, Wind River and XtratuM hypervisors. All these projects use or can use paravirtualization techniques to support high performance virtual machines on x86 hardware by implementing a virtual machine that does not implement the hard-to-virtualize parts of the actual x86 instruction set.

A hypervisor provides the virtualization of the underlying computer system. In full virtualization, a guest operating system runs unmodified on a hypervisor. However, improved performance and efficiency is achieved by having the guest operating system communicate with the hypervisor. By allowing the guest operating system to indicate its intent to the hypervisor, each can cooperate to obtain better performance when running in a virtual machine. This type of communication is referred to as paravirtualization.

In 2005, VMware proposed a paravirtualization interface, the Virtual Machine Interface (VMI), as a communication mechanism between the guest operating system and the hypervisor. This interface enabled transparent paravirtualization in which a single binary version of the operating system can run either on native hardware or on a hypervisor in paravirtualized mode.

That reminds me, that also the Support in the kernel of it, has been dropped far in the past around 2011, while Starting in '08 VirtIO was introduced (maybe as a successor)

And an IMHO:

Similar to the TCP/IPv4 stack, it represents a legacy technology implemented with the intention of enduring indefinitely, even though future developments were unforeseen at the time of its deployment. I hope I did not strain your eyes too much with this extended answer. ;)

Reference(s):

• https://en.wikipedia.org/wiki/Paravirtualization
• https://docs.kernel.org/driver-api/virtio/virtio.html#:~:text=Virtio%20is%20an%20open%20standard,spec%20(%5B1%5D).
• https://docs.oasis-open.org/virtio/virtio/v1.2/virtio-v1.2.html
• https://www.redhat.com/en/blog/virtqueues-and-virtio-ring-how-data-travels
• A lot out of my brand
• and also heavy used a well-known search engine ;)

Answer 4

Another answer contains:

Now, you could pass through a physical network interface from the host system, but that has a number of issues:

• It requires actually having a usable network interface for each VM. This obviously means that anybody on a laptop would be out of luck, but it also means that most large scale VM hosting providers would be out of luck as well (they may be running many dozens of VMs on the same host).

As a counter-argument, Single Root IO Virtualization (SR-IOV) can be used to allow a physical PCIe device to present itself multiple times through the PCIe bus. From the NVIDIA (ex Mellanox) page linked above:

This technology enables multiple virtual instances of the device with separate resources. Mellanox adapters are capable of exposing up to 127 virtual instances (Virtual Functions (VFs) for each port in the Mellanox ConnectX® family cards. These virtual functions can then be provisioned separately. Each VF can be seen as an additional device connected to the Physical Function. It shares the same resources with the Physical Function, and its number of ports equals those of the Physical Function. SR-IOV is commonly used in conjunction with an SR-IOV enabled hypervisor to provide virtual machines direct hardware access to network resources hence increasing its performance.

As one example of a VM hosting provider, the Amazon Web Services Enhanced networking on Linux page contains:

Enhanced networking uses single root I/O virtualization (SR-IOV) to provide high-performance networking capabilities on supported instance types. SR-IOV is a method of device virtualization that provides higher I/O performance and lower CPU utilization when compared to traditional virtualized network interfaces. Enhanced networking provides higher bandwidth, higher packet per second (PPS) performance, and consistently lower inter-instance latencies. There is no additional charge for using enhanced networking.