0

We use qemu virtualization with Linux hosts for many years now! It's a great tool! We use it both for Linux VM’s and Windows VM’s!

But we have a strange unsolved performance issue with all the VM’s running Windows 10 or Windows Server 2016: Very slow I/O when working (copy, delete) with small (under 100KB) files - Under 1MB/s throughput rate. Throughput with big files is 35 Mb/s maximum, still below what I/O controller is capable of.

The problem arises with any Storage formats: raw, qcow.

Very slow I/O on Windows Server 2016 VM! Screenshot of a file copy operation

Running Win10 or Win Server 2016 on the same hardware but directly installed...no problems!

If we try running on the same hosts VM’s with Windows 7 and Windows Server 2012 we have great I/O speeds on the same files, at least 30MB/s throughput! Throughput with big files is at least 60MB/s.

What could be wrong? When using raw storage devices we use the latest virtio drivers from fedora link: https://docs.fedoraproject.org/en-US/quick-docs/creating-windows-virtual-machines-using-virtio-drivers/index.html

Virtio driver details on a Windows Server 2016 VM

Same issue appears on 4 different hosts! The hardware setup:

Host Centos 7 Machine (CentOS Linux release 7.3.1611 (Core)) : HP ML 380 G8 Server, 192GB Ram, 2 Xeon Processors, SAS Storage.

VM’s: KVM virtualization, Windows 10 version 17134.590 (at least 4GB of RAM per VM) or Windows Server 2016 version 1607-Build 14393.2273 (at least 10GB of RAM per VM)

Below the Virsh setup of a VM maybe somebody can spot a problem!

We think it’s a driver issue but we got no clue on where to find virtio drivers that work with the mentioned editions of Windows.

Any ideas? Thanks!

KVM Domain Description

<domain type='kvm' id='9'>
  <name>xxxx</name>
  <uuid>yyyy</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/KVM/xxx.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <boot order='3'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <backingStore/>
      <target dev='sda' bus='sata'/>
      <readonly/>
      <boot order='1'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/root/Desktop/ESD_ISO.iso'/>
      <backingStore/>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='sata0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:38:9f:70'/>
      <source network='NAT' bridge='virbr0'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='spice' port='5902' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <stats period='5'/>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>
  • You could explicitely set the iothreads number per controller in this xmldump and see if it helps. Not the clear solution, but a wild guess at this time. You also didn't mention the %busy disk liad on your host or SAN system - small files i/o is very iops-expensive. – drookie Feb 23 at 5:50
  • I cannot use iothreads option for the VM. My qemu version does not allow it. IO usage on host is low: under 10% busy. Even in virt-top we see very little IO usage. I think it's a driver problem in the VM or an option not properly configured in the domain for this guests. However, as I said, no such problems when runnning Win 7 or Win Server 2012 Guests. – Enovator Feb 23 at 8:57
  • You can, if you install the EV version of Qemu/KVM from Qemu SIG. – drookie Feb 23 at 10:48
  • Basically you're saying Qemu SIG has better Win 10 and Windows Server 2016 support? Or this is just a guess? I can try to setup a test... – Enovator Feb 23 at 14:36
  • Basically I say Qemu EV contains more recent version of Qemu/KVM which has the iothreads support that we use in production with Windows 10 guests, and I haven’t seen the problems you’re experienced, thus it may be related to iothreads or simply to version. We also use q35 VMs, not fx440 ones. – drookie Feb 23 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.