1

I am running Manjaro KDE and I have managed to setup a Windows 10 VM with GPU passthrough for Windows things and it was working great until I noticed a weird behavior. When I try to reboot the VM, or even randomly (I left it on over night and in the morning it happened), the VM switches its state from Running to Pause. If I try to resume I get this error:

Error unpausing domain: internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1364, in resume
    self._backend.resume()
  File "/usr/lib/python3.9/site-packages/libvirt.py", line 2175, in resume
    if ret == -1: raise libvirtError ('virDomainResume() failed', dom=self)
libvirt.libvirtError: internal error: unable to execute QEMU command 'cont': Resetting the Virtual Machine is required

In order to get it going again, I have to Force off then start again. I tried to Google this error but the results are inconclusive. Some reddit post mentioned ioapic but it didn’t say where to add it or how.

Can anyone help me figure this out please? Here is the VM xml.

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>windows10</name>
  <uuid>baab6e03-8e1c-4275-bbc8-ea1e390cd0d2</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16777216</memory>
  <currentMemory unit="KiB">16777216</currentMemory>
  <vcpu placement="static">8</vcpu>
  <iothreads>2</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="8"/>
    <vcpupin vcpu="1" cpuset="9"/>
    <vcpupin vcpu="2" cpuset="10"/>
    <vcpupin vcpu="3" cpuset="11"/>
    <vcpupin vcpu="4" cpuset="12"/>
    <vcpupin vcpu="5" cpuset="13"/>
    <vcpupin vcpu="6" cpuset="14"/>
    <vcpupin vcpu="7" cpuset="15"/>
    <emulatorpin cpuset="0-1"/>
    <iothreadpin iothread="1" cpuset="0-1"/>
    <iothreadpin iothread="2" cpuset="2-3"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-5.1">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/windows10_VARS.fd</nvram>
    <bootmenu enable="no"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <synic state="on"/>
      <stimer state="on"/>
      <reset state="on"/>
      <vendor_id state="on" value="1234567890ab"/>
      <frequencies state="on"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on">
    <topology sockets="1" dies="1" cores="4" threads="2"/>
    <cache mode="passthrough"/>
    <feature policy="require" name="topoext"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" present="no" tickpolicy="catchup"/>
    <timer name="pit" present="no" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
    <timer name="tsc" present="yes" mode="native"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native"/>
      <source dev="/dev/disk/by-id/ata-ST2000DX001-1CM164_Z1E83TP0"/>
      <target dev="vdb" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/data/vmstorage/windows10.qcow2"/>
      <target dev="vdc" bus="virtio"/>
      <boot order="1"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </disk>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native"/>
      <source dev="/dev/disk/by-id/ata-Samsung_SSD_840_PRO_Series_S1ATNSAF534885M"/>
      <target dev="vdd" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x0a" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/lucian/Downloads/virtio-win-0.1.189.iso"/>
      <target dev="sda" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x9"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0xa"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0xb"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x3"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0xc"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x4"/>
    </controller>
    <controller type="pci" index="12" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x0b" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0xd"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x5"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0xe"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x6"/>
    </controller>
    <controller type="pci" index="15" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="15" port="0xf"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x7"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </controller>
    <interface type="direct">
      <mac address="52:54:00:20:dd:64"/>
      <source dev="enp4s0" mode="bridge"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <input type="mouse" bus="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0e" function="0x0"/>
    </input>
    <input type="keyboard" bus="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0f" function="0x0"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
    </graphics>
    <video>
      <model type="virtio" heads="1" primary="yes">
        <acceleration accel3d="yes"/>
      </model>
      <address type="pci" domain="0x0000" bus="0x0d" slot="0x00" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x0781"/>
        <product id="0xcfc9"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x08" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value="-object"/>
    <qemu:arg value="input-linux,id=mouse1,evdev=/dev/input/by-id/usb-Corsair_Corsair_M65_Gaming_Mouse-event-mouse"/>
    <qemu:arg value="-object"/>
    <qemu:arg value="input-linux,id=kbd1,evdev=/dev/input/by-id/usb-SINO_WEALTH_USB_KEYBOARD-event-kbd,grab_all=on,repeat=on"/>
  </qemu:commandline>
</domain>

EDIT:

I have added the contents of the log after the above error appears:

LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/snapd/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-1-windows10 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-windows10/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-windows10/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-windows10/.config \
QEMU_AUDIO_DRV=spice \
/usr/bin/qemu-system-x86_64 \
-name guest=windows10,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-windows10/master-key.aes \
-blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/windows10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-5.1,accel=kvm,usb=off,vmport=off,dump-guest-core=off,kernel_irqchip=on,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=pc.ram \
-cpu host,migratable=on,topoext=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff,hv-vpindex,hv-synic,hv-stimer,hv-reset,hv-vendor-id=1234567890ab,hv-frequencies,kvm=off,host-cache-info=on,l3-cache=off \
-m 16384 \
-object memory-backend-ram,id=pc.ram,size=17179869184 \
-overcommit mem-lock=off \
-smp 8,sockets=1,dies=1,cores=4,threads=2 \
-object iothread,id=iothread1 \
-object iothread,id=iothread2 \
-uuid baab6e03-8e1c-4275-bbc8-ea1e390cd0d2 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot menu=off,strict=on \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
-device pcie-root-port,port=0x8,chassis=7,id=pci.7,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=9,id=pci.9,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0xb,chassis=10,id=pci.10,bus=pcie.0,addr=0x1.0x3 \
-device pcie-root-port,port=0xc,chassis=11,id=pci.11,bus=pcie.0,addr=0x1.0x4 \
-device pcie-pci-bridge,id=pci.12,bus=pci.11,addr=0x0 \
-device pcie-root-port,port=0xd,chassis=13,id=pci.13,bus=pcie.0,addr=0x1.0x5 \
-device pcie-root-port,port=0xe,chassis=14,id=pci.14,bus=pcie.0,addr=0x1.0x6 \
-device pcie-root-port,port=0xf,chassis=15,id=pci.15,bus=pcie.0,addr=0x1.0x7 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.8,addr=0x0 \
-blockdev '{"driver":"host_device","filename":"/dev/disk/by-id/ata-ST2000DX001-1CM164_Z1E83TP0","aio":"native","node-name":"libvirt-4-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-4-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-4-storage"}' \
-device virtio-blk-pci,bus=pci.2,addr=0x0,drive=libvirt-4-format,id=virtio-disk1,write-cache=on \
-blockdev '{"driver":"file","filename":"/data/vmstorage/windows10.qcow2","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-device virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-3-format,id=virtio-disk2,bootindex=1 \
-blockdev '{"driver":"host_device","filename":"/dev/disk/by-id/ata-Samsung_SSD_840_PRO_Series_S1ATNSAF534885M","aio":"native","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-device virtio-blk-pci,bus=pci.10,addr=0x0,drive=libvirt-2-format,id=virtio-disk3,write-cache=on \
-blockdev '{"driver":"file","filename":"/home/lucian/Downloads/virtio-win-0.1.189.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.0,drive=libvirt-1-format,id=sata0-0-0 \
-netdev tap,fd=34,id=hostnet0,vhost=on,vhostfd=35 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:20:dd:64,bus=pci.1,addr=0x0 \
-device virtio-mouse-pci,id=input0,bus=pcie.0,addr=0xe \
-device virtio-keyboard-pci,id=input1,bus=pcie.0,addr=0xf \
-spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on \
-device virtio-vga,id=video0,virgl=on,max_outputs=1,bus=pci.13,addr=0x0 \
-device usb-host,hostdevice=/dev/bus/usb/002/003,id=hostdev0,bus=usb.0,port=1 \
-device vfio-pci,host=0000:08:00.0,id=hostdev1,bus=pci.3,addr=0x0 \
-device vfio-pci,host=0000:08:00.1,id=hostdev2,bus=pci.6,addr=0x0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 \
-object input-linux,id=mouse1,evdev=/dev/input/by-id/usb-Corsair_Corsair_M65_Gaming_Mouse-event-mouse \
-object input-linux,id=kbd1,evdev=/dev/input/by-id/usb-SINO_WEALTH_USB_KEYBOARD-event-kbd,grab_all=on,repeat=on \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-02-17 20:44:07.568+0000: Domain id=1 is tainted: high-privileges
2021-02-17 20:44:07.568+0000: Domain id=1 is tainted: custom-argv
2021-02-17T20:44:11.349338Z qemu-system-x86_64: vfio_err_notifier_handler(0000:08:00.1) Unrecoverable error detected. Please collect any data possible and then kill the guest
2021-02-17T20:44:11.349472Z qemu-system-x86_64: vfio_err_notifier_handler(0000:08:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest
2021-02-17T20:46:53.389263Z qemu-system-x86_64: vfio_err_notifier_handler(0000:08:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest
2021-02-17T20:46:53.411839Z qemu-system-x86_64: vfio_err_notifier_handler(0000:08:00.1) Unrecoverable error detected. Please collect any data possible and then kill the guest
2021-02-17T20:47:15.188142Z qemu-system-x86_64: terminating on signal 15 from pid 1105 (/usr/bin/libvirtd)
2021-02-17 20:47:16.991+0000: shutting down, reason=destroyed
6
  • Check the virtual machine log in /var/log/libvirt/qemu Feb 17 '21 at 17:39
  • I have added the contents of the log (the last part after the last run). I have also found this site and tried to add some fixes and optimizations (I have also changed the xml section to reflect the new changes).
    – lucian
    Feb 17 '21 at 20:53
  • The unrecoverable errors relate to the two physical devices you have passed through to the guest. That's about all I can tell from this log. Feb 17 '21 at 21:05
  • and are those related to the unable to execute QEMU command 'cont' part?
    – lucian
    Feb 17 '21 at 21:10
  • Yes, an "Unrecoverable error" would be why it cannot 'cont'inue. Feb 17 '21 at 22:40
1

After digging around for a solution, I had another look at the kernel parameters that I was passing at boot in `/etc/default/grub'. The relevant line in this file is this one

GRUB_CMDLINE_LINUX_DEFAULT="quiet apparmor=1 security=apparmor udev.log_priority=3 amd_iommu=on iommu=pt pcie_ports=compat pci=nommconf"

Initially, I had no setting for the pcie_ports and I believe auto is used as the default. Reading up on this I noticed that there are two more options: native (which appears in a few tutorials I have stumbled upon, but it doesn't solve my problem) and compat.

Setting pcie_ports=compat solved the restart and shutdown problem with my Windows 10 VM.

The pci=nommconf I have set because I have a very old HP printer that has an integrated card reader which generates a lot of errors in dmesg and it is very unstable in windows. With this, I get a bit more use out of this printer.

I am unsure what are the consequences of pcie_ports=compat on my host OS! So I am curious if anyone can shine a bit more light on this issue!

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.