Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a Qemu-KVM host system setup on CentOS 6.3. Four 1TB SATA HDDs working in Software RAID10. Guest CentOS 6.3 is installed on separate LVM. People say that they see guest performance almost equal to host performance, but I don't see that. My i/o tests are showing 30-70% slower performance on guest than on host system. I tried to change scheduler (set elevator=deadline on host and elevator=noop on guest), set blkio.weight to 1000 in cgroup, change io to virtio... But none of these changes gave me any significant results. This is a guest .xml config part:

<disk type='file' device='disk'>
  <driver name='qemu' type='raw'/>
  <source file='/dev/vgkvmnode/lv2'/>
  <target dev='vda' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>

There are my tests:

Host system:

iozone test

# iozone -a -i0 -i1 -i2 -s8G -r64k
                                                            random  random 
              KB  reclen   write rewrite    read    reread    read   write 
         8388608      64  189930  197436   266786   267254   28644   66642 

dd read test: one process and then four simultaneous processes

# dd if=/dev/vgkvmnode/lv2 of=/dev/null bs=1M count=1024 iflag=direct
1073741824 bytes (1.1 GB) copied, 4.23044 s, 254 MB/s

# dd if=/dev/vgkvmnode/lv2 of=/dev/null bs=1M count=1024 iflag=direct skip=1024 & dd if=/dev/vgkvmnode/lv2 of=/dev/null bs=1M count=1024 iflag=direct skip=2048 & dd if=/dev/vgkvmnode/lv2 of=/dev/null bs=1M count=1024 iflag=direct skip=3072 & dd if=/dev/vgkvmnode/lv2 of=/dev/null bs=1M count=1024 iflag=direct skip=4096
1073741824 bytes (1.1 GB) copied, 14.4528 s, 74.3 MB/s
1073741824 bytes (1.1 GB) copied, 14.562 s, 73.7 MB/s
1073741824 bytes (1.1 GB) copied, 14.6341 s, 73.4 MB/s
1073741824 bytes (1.1 GB) copied, 14.7006 s, 73.0 MB/s

dd write test: one process and then four simultaneous processes

# dd if=/dev/zero of=test bs=1M count=1024 oflag=direct
1073741824 bytes (1.1 GB) copied, 6.2039 s, 173 MB/s

# dd if=/dev/zero of=test bs=1M count=1024 oflag=direct & dd if=/dev/zero of=test2 bs=1M count=1024 oflag=direct & dd if=/dev/zero of=test3 bs=1M count=1024 oflag=direct & dd if=/dev/zero of=test4 bs=1M count=1024 oflag=direct
1073741824 bytes (1.1 GB) copied, 32.7173 s, 32.8 MB/s
1073741824 bytes (1.1 GB) copied, 32.8868 s, 32.6 MB/s
1073741824 bytes (1.1 GB) copied, 32.9097 s, 32.6 MB/s
1073741824 bytes (1.1 GB) copied, 32.9688 s, 32.6 MB/s

Guest system:

iozone test

# iozone -a -i0 -i1 -i2 -s512M -r64k
                                                            random  random
              KB  reclen   write rewrite    read    reread    read   write
          524288      64   93374  154596   141193   149865   21394   46264 

dd read test: one process and then four simultaneous processes

# dd if=/dev/mapper/VolGroup-lv_home of=/dev/null bs=1M count=1024 iflag=direct skip=1024
1073741824 bytes (1.1 GB) copied, 5.04356 s, 213 MB/s

# dd if=/dev/mapper/VolGroup-lv_home of=/dev/null bs=1M count=1024 iflag=direct skip=1024 & dd if=/dev/mapper/VolGroup-lv_home of=/dev/null bs=1M count=1024 iflag=direct skip=2048 & dd if=/dev/mapper/VolGroup-lv_home of=/dev/null bs=1M count=1024 iflag=direct skip=3072 & dd if=/dev/mapper/VolGroup-lv_home of=/dev/null bs=1M count=1024 iflag=direct skip=4096
1073741824 bytes (1.1 GB) copied, 24.7348 s, 43.4 MB/s
1073741824 bytes (1.1 GB) copied, 24.7378 s, 43.4 MB/s
1073741824 bytes (1.1 GB) copied, 24.7408 s, 43.4 MB/s
1073741824 bytes (1.1 GB) copied, 24.744 s, 43.4 MB/s

dd write test: one process and then four simultaneous processes

# dd if=/dev/zero of=test bs=1M count=1024 oflag=direct
1073741824 bytes (1.1 GB) copied, 10.415 s, 103 MB/s

# dd if=/dev/zero of=test bs=1M count=1024 oflag=direct & dd if=/dev/zero of=test2 bs=1M count=1024 oflag=direct & dd if=/dev/zero of=test3 bs=1M count=1024 oflag=direct & dd if=/dev/zero of=test4 bs=1M count=1024 oflag=direct
1073741824 bytes (1.1 GB) copied, 49.8874 s, 21.5 MB/s
1073741824 bytes (1.1 GB) copied, 49.8608 s, 21.5 MB/s
1073741824 bytes (1.1 GB) copied, 49.8693 s, 21.5 MB/s
1073741824 bytes (1.1 GB) copied, 49.9427 s, 21.5 MB/s

I wonder is that normal situation or did I missed something?

share|improve this question
    
You said you have changed your guest to use a bus type of virtio which does give better performance, but it must have the virtio drivers installed to reap these benefits. Youdidn't say ifyou are using those or not. I don't know CentOS thoroughly enough to comment on wheather these drivers will be present in your guest by default or not. –  jwbensley Sep 8 '12 at 19:37
    
@javano CentOS always includes virtio and you would have to explicitly remove it by rebuilding the kernel packages. –  Michael Hampton Sep 8 '12 at 21:50
    
Always handy to know, cheers :) –  jwbensley Sep 9 '12 at 14:18
add comment

1 Answer

up vote 9 down vote accepted

You're not done with performance tuning yet.

  <driver name='qemu' type='raw' cache='writethrough' io='native'/>

First is which I/O mechanism to use.

QEMU has two asynchronous I/O mechanisms: POSIX AIO emulation using a pool of worker threads and native Linux AIO.

Set either io='native' or io='threads' in your XML to benchmark each of these.

Second is which caching mechanism to use. You can set cache='writeback', cache='writethrough' or you can turn it off with cache='none', which you actually may find works best.

If you're using raw volumes or partitions, it is best to avoid the cache completely, which reduces data copies and bus traffic.

Don't use writeback unless your RAID array is battery-backed, or you risk losing data. (Of course, if losing data is OK, then feel free.)

Third, some other things that may help include turning off barriers, and using the deadline scheduler in the guest.

Finally, do some research. IBM made a very interesting presentation on KVM I/O performance at the 2010 Linux Plumbers Conference. In addition they have an extensive set of best practices on using KVM which will certainly be of interest.

P.S. Lengthy sequential reads and writes are rarely representative of a real-world workload. Try doing benchmarks with other types of workloads, ideally the actual application(s) you intend to run in production.

share|improve this answer
    
+1 for "test with your IO pattern" –  Javier Sep 8 '12 at 23:59
    
Oh, good likes to the IBM docs, +1 :) –  jwbensley Sep 9 '12 at 14:17
1  
Very helpful, thanks! Now I'm improved my guest results up to 90% from host performance. My fault was quite stupid: virsh reset <domain> didn't apply my virsh edit <domain> changes, and I was believing that guest used virtio, but it actually didn't. Only virsh destroy <domain> followed by virsh start <domain> helped. Virtio rules! ;) –  Evolver Sep 9 '12 at 15:47
    
cache=writeback does not add any real-life risks (important data is not in danger, only in-flight data, which is discarded on crash anyways). Only cache=unsafe does. Writeback does not imply additional hardware requirements ("battery backed RAID array" does not help in any way). It has the same integrity level as an HDD write-cache: both are flushed when necessary by the operating system. –  korkman May 15 '13 at 15:51
    
io='native' gave almost 20-30% extra WRITE performance in my case –  rahul286 Jul 10 '13 at 13:39
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.