0

I have to NIC devices on the host:

# list Ethernet PCI devices to find out names
lspci -nn | grep Ethernet
# 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)
# 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

# to get <domain>.<bus>.<slot>.<function>
lspci -n -s 0000:04:00.0
# 04:00.0 0200: 10ec:8168 (rev 06)

And I want to pass through the device 0000:04:00.0 to the KVM ubuntu 20.04 virtual machine. The host may not see this device when VM is running. To bind PCI NIC to the guest I successfully followed instruction VFIO - "Virtual Function I/O". But unbinding from the guest to the host came more difficult.

I bind NIC device from the host to the guest manually in the following way:

# find Ethernet controller
ls -l /sys/class/net/ | grep pci
# enp3s0 -> ../../devices/pci0000:00/0000:00:1c.5/0000:03:00.0/net/enp3s0
# enp4s0 -> ../../devices/pci0000:00/0000:00:1c.7/0000:04:00.0/net/enp4s0
lspci -nn | grep Ethernet
# 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)
# 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

# check the IOMMU grouping
for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort | grep '04:00\|05:00'
# /sys/kernel/iommu_groups/12/devices/0000:04:00.0
# /sys/kernel/iommu_groups/13/devices/0000:05:00.0

# find MAC addresses
for eth in enp4s0 enp5s0; do ifconfig $eth | grep ether | echo "MAC" "$eth:" $(awk '{print $2}'); done
# MAC enp4s0: 50:...
# MAC enp5s0: 18:...

# load the vfio-pci device driver
modprobe vfio_pci

# unbind from host
echo 0000:04:00.0 | sudo tee /sys/bus/pci/devices/0000:04:00.0/driver/unbind
# make available for guests
echo 10ec 8168 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

# IOMMU group for device 0000:04:00.0 is 12
# add libvirt-qemu to sudoers
sudo usermod -aG sudo libvirt-qemu
sudo chown libvirt-qemu:libvirt-qemu /dev/vfio/12

# look at what other devices are in the group to free it for use by VFIO
ls -l /sys/bus/pci/devices/0000:04:00.0/iommu_group/devices | awk '{print $9 $10 $11}'
# 0000:04:00.0->../../../../devices/pci0000:00/0000:00:1c.5/0000:04:00.0
# also add other devices if they belong to the same group (not needed in this case)

# make sure the device is not visible from host
ls -l /sys/class/net/ | grep enp4s0

Then I created new Ubuntu 20.04 VM using Virtual Machine Manager (virt-manager) to run on KVM.

I added new device to VM by editing its xml configuration in virt-manager during creation. In particular, <devices> section contains the following tag

 <hostdev mode="subsystem" type="pci" managed="yes">
   <driver name="vfio"/>
   <source>
     <address domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
   </source>
   <mac address='50:...'/>
   <address type="pci"> 
     <zpci uid="0x0001" fid="0x00000000"/>
   </address>
 </hostdev>

Than I installed Ubuntu 20.04 in regular way. The system reboots properly, without deadlocks (black screen).

When I turn off the VM, I want to return the PCI NIC to the host. I did a research on forums but there is no clear instructions on how to do that.

If I reboot the host, all devices return to the host, so vfio binding is released. But how I can do that without host rebooting.

2 Answers 2

1

To do it manually, you can use the same way that you unbound it from the host to make it available to the VM.

Something like this should do

echo 0000:04:00.0 | sudo tee /sys/bus/pci/devices/0000:04:00.0/driver/unbind
echo 10ec 8168 | sudo tee /sys/bus/pci/drivers/<host-driver-here>/new_id

You need to know what is the host driver you want to rebind the device to, in order to do that you could use

 lspci -n -v -s 0000:04:00.0

After a fresh reboot, and look for the "Kernel driver in use:" line, you will have the name of the driver there

0

libvirt does the necessary unbinding and rebinding for you, and apparently it also restores the previous state on VM shutdown.

If you don't do anything to the driver bindings prior to VM start, it should do exactly what you want.

2
  • If I pass through NIC to the guest via virt-manager than it appears on the guest. But the host also sees these device and shows Ethernet connection in the system tray as well. I need completely hide it on the host. So, the question remains: how to rebind NIC to the host manually, after the guest shutdown.
    – Alexander
    Oct 15, 2021 at 10:51
  • Ah, so the device reappearing is the problem. Oct 15, 2021 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.