I have two very similar x86_64 machines running the same kernel (4.7.8), and the same version of libvirt. The only difference between the two machines is the amount of memory they have, machine A has 8G while machine B has 32G, and the BIOS version running (more details below).

On machine A I can run guests without any problems, but on machine B it happens very often that a guest will get into paused state while it is booting (it is always during the booting stage), and I won't be able to resume it. This happens quite often, and even more often when two guests are started, one after the other, when frequently one or both guests will get into that paused state. This issue NEVER happens on machine A.

Looking at the guest's log I can see that Suberror 3 occurs when the issue is seen:

2020-05-02 16:38:16.949+0000: starting up libvirt version: 1.2.20, qemu version: 2.6.0
LC_ALL=C PATH=/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name testvm -S -machine pc-1.1,accel=kvm,usb=off -cpu Broadwell,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid e1e10e42-55f5-4927-aa31-71114d9edc33 -smbios type=1,product=QEMU -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-testvm/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,clock=vm,driftfix=slew -no-shutdown -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -drive file=/mnt/disk0/.libvirt/ubmtest_0.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=directsync,aio=native -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -netdev tap,fd=18,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:01:fa:01:51,bus=pci.0,addr=0x3 -netdev tap,fd=19,id=hostnet1 -device e1000,netdev=hostnet1,id=net1,mac=52:54:02:fa:02:51,bus=pci.0,addr=0x6 -netdev tap,fd=20,id=hostnet2 -device e1000,netdev=hostnet2,id=net2,mac=52:54:0b:fa:0b:51,bus=pci.0,addr=0x7 -netdev tap,fd=21,id=hostnet3 -device e1000,netdev=hostnet3,id=net3,mac=52:54:64:fa:64:51,bus=pci.0,addr=0x8 -netdev tap,fd=22,id=hostnet4 -device e1000,netdev=hostnet4,id=net4,mac=52:54:c8:fa:c8:51,bus=pci.0,addr=0x9 -netdev tap,fd=23,id=hostnet5 -device e1000,netdev=hostnet5,id=net5,mac=52:54:d3:fa:d3:51,bus=pci.0,addr=0xb -netdev tap,fd=24,id=hostnet6 -device e1000,netdev=hostnet6,id=net6,mac=52:54:dd:fa:dd:51,bus=pci.0,addr=0xa -netdev tap,fd=25,id=hostnet7 -device e1000,netdev=hostnet7,id=net7,mac=52:54:ff:fa:ff:51,bus=pci.0,addr=0xc -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
Domain id=1 is tainted: high-privileges
char device redirected to /dev/pts/0 (label charserial0)
KVM internal error. Suberror: 3
extra data[0]: 8000003e
extra data[1]: 31
RAX=0000000000000000 RBX=ffffffff81f38280 RCX=0000000000000000 RDX=0000000000000000
RSI=0000000000000000 RDI=0000000000000000 RBP=ffffffff81e03e98 RSP=ffffffff81e03e98
R8 =ffff88001fc0dd80 R9 =0000000000000000 R10=00000000fffee09f R11=0000000000037000
R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=ffffffff81e00000
RIP=ffffffff810645d6 RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 0000000000000000 ffffffff 00000000
CS =0010 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0018 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0000 0000000000000000 ffffffff 00000000
FS =0000 0000000000000000 ffffffff 00000000
GS =0000 ffff88001fc00000 ffffffff 00000000
LDT=0000 0000000000000000 000fffff 00000000
TR =0040 ffff88001fc142c0 00002087 00008b00 DPL=0 TSS64-busy
GDT=     ffff88001fc09000 0000007f
IDT=     ffffffffff574000 00000fff
CR0=80050033 CR2=00007f1cb3691037 CR3=00000000167a7000 CR4=003406f0
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000fffe0ff0 DR7=0000000000000400
Code=89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84 00 00 00 00 00 55 49 89 c9

As I mentioned before, all the software on both of the systems are identical, and most of the hardware as well.

Here are some relevant specs shared by both guests:

  • CPU: Intel(R) Core(TM) i5-5350U CPU @ 1.80GHz
  • Kernel: 4.7.8
  • libvirt 1.2.20
  • QEMU 2.6.0

Relevant differences:

  • Machine A BIOS: American Megatrends Inc. v0.04 06/16/2016 release 5.6
  • Machine B BIOS: American Megatrends Inc. v0.02 07/07/2015 release 5.6

Does anyone have any clues to what might be happening here? Is it possible that this is related to the different BIOS versions?


  • 1
    What is the physical hardware? Why is your clock wrong? – Michael Hampton Nov 19 '18 at 15:36
  • Hardware is a custom made board, and to be honest I don't know much about it, but if there is anything specific about the board that it is useful to know I can ask here... Clock is wrong because the test team has being doing some overnight tests with this machine, and this included changing the clock. – igalvez Nov 20 '18 at 9:58
  • Something is different between the two boards. Perhaps one is defective. Crashes like this almost always have two possible causes: a defective system board/CPU/RAM, or some bit wasn't read correctly from the disk. The defective hardware seems much more likely to me. – Michael Hampton Nov 20 '18 at 13:12

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.