4

I have been battling with a kvm setup under debian for two weeks now, specifically with guest I/O disk performance.

The system:

-Supermicro 1018d-73mtf (X10SL7-F motherboard)
-16GB ECC/UB
-Intel Xeon E3-1240v3
-6xWD Red 750GB 6Gb/s

On this i am running a Debian Wheezy on two of the disks, the other four disks are setup with md for raid5 with LVM on top for guest storage. Performance directly on the raid5 (measured by creating a LV and mounting it and running bonnie++ and dd tests) is fine, giving me ~220/170MB/s read/write, but on guests i get decent reads and 40-50MB/s writes, tested on both Windows (Server 2012) and Linux (Debian). I have read up on aligning disks and partitions and recreated the raid and lvm setup by the book, but havnt recieved any performance boosts.

When measuring loads with atop during writes directly from the host i can see that the disks and lvm are getting high loads, but measuring while a guest is writing shows the disks at ~20-30% and lvm getting "red" (100%).

The normal tweaks of KVM/host system have been done, setting scheduler to deadline, setting stripe caches for the raid, cache=none on the guests, reflashing the SAS controller card to IT-mode (LSI 2308) and i am out of ideas, here is a pastebin with relevant information about the setup in the hopes of someone noticing something i've done wrong http://pastebin.com/vuykxeVg.

If you need anything else il paste it.

Edits:

This is basicly how the drives, md and lvm is set up, with some changes because i am running 3 disks + spare. http://dennisfleurbaaij.blogspot.se/2013/01/setting-up-linux-mdadm-raid-array-with.html

Screenshots of atop during host and guest writetests (bonnie++)

Host: https://i.stack.imgur.com/sHBuK.png

Guest: https://i.stack.imgur.com/OFv10.png

12
  • Can you show me the output of cat /sys/block/sda/queue/scheduler?
    – Chris
    Dec 3, 2013 at 14:52
  • sda? Sure but its not used for the guests: noop deadline [cfq] For the actual raid-disks its: noop [deadline] cfq
    – FrontSlash
    Dec 3, 2013 at 15:02
  • I meant you should replace sda with your internal disk inside the guest.
    – Chris
    Dec 3, 2013 at 15:28
  • On the linux guest i have been running noop, just did another bonnie++ test and got slightly better performance than i did previously on the linux vm, from 50 to 65MB/s seq writes, without changing anything
    – FrontSlash
    Dec 3, 2013 at 18:56
  • What is RAID(5?!) doing inside VM o_O? (Following snapshot link for Guest)
    – Veniamin
    Dec 4, 2013 at 18:29

4 Answers 4

2

Not sure that my note covers all the problem but with this storage configuration you could not get proper alignment.

Lets see,

  • You can align partition boundaries according to RAID stripe size, it is OK.
  • You can set file-system optimization parameters accordingly, it is also OK.

  • But in order to align LVM properly you need the RAID stripes to fit LVM extent.

LVM extent size is always power of 2. Thus your RAID stripe size needs to be power of 2. In order to get it you need a number of disks in RAID5 to be equal to 2^N+1 = 3, 5, 9 ...

With 4 disks in RAID5 it is impossible.

Since software RAID5 has no protected write-back cache It may significantly vulnerable with "partial stripe write penalty".

May be you also have another causes restricting write performance but the first I would have done - migration to RAID10. With RAID10 on all 6 disks you are able to get read performance and guest storage capacity comparable to your initial setup ... and no headache with alignment ;).

7
  • Mind you that only 3 of the disks are active in the raid5, the fourth is spare, as seen in the mdstat output
    – FrontSlash
    Dec 3, 2013 at 15:00
  • Ok, did you perform partition aligning and FS optimization inside VM?
    – Veniamin
    Dec 3, 2013 at 15:54
  • Yep, see the pastebin it has parted info about the windows part
    – FrontSlash
    Dec 3, 2013 at 15:58
  • Windows is not aware that your stripe is 1MB, nor is virtio driver. I would try RAID10 at least to finaly commit the issue with partial stripe write.
    – Veniamin
    Dec 3, 2013 at 16:34
  • My stripe isnt 1MB, i've done the same tests with a raid1 mounted in windows and debian with no improvement, might that you missed that i have tested this with both linux and windows, linux with properly aligned sectors (mkfs.ext4 –m 0 -b 4096 -E stripe-width=256,stride=128) Stripe-width based on the fact that there are two data disks and stride being 128 since stripe-size/block-size.
    – FrontSlash
    Dec 3, 2013 at 18:47
1

I've done extensive testing on KVM and caching performance (you can read here, here and here) and many of the recommendation that you find on the Internet is obsolete of plain wrong. But let's proceed one step at a time...

  1. RAID5 needs a much smaller chunk size (in the order of 32-64K) than yours (512K, as seen from pastebin) and a BBU (for writeback caching) to be viable for anything different that sequential read/write patterns.
  2. Avoid RAID5 for any workloads that need IOPS and/or issues a significant portion of random writes. This is EXACTLY what a guest OS need (IOPS and random write speed), so the choice of RAID5 (without and hardware BBU for enabling array-wide writeback caching) is the wrong one. Use RAID10 instead.
  3. Absolutely enable cache=writeback for your guests. Cache=none is slightly better only on sequential reads/writes or absolutely irregular and cache-unfriendly patterns.
  4. Use virtio drivers inside your guests each time it is possible.
0

If you google for kvm disk io performance there is an amazing number of hits. So I will stay with XEN. ;-)

So this seem to be a general problem with KVM.

The common mainline is to use the right virtio-driver.

Then there are two lines:

  • use caching on the KVM-server

  • do not use caching in the clients on a KVM-server if the hostOS is already doing disk caching. This applies to Xen and the other hypervisors too.

Both seem to work - but both say that the defaults are just awful.

7
  • Using latest virtio, cache settings have all been tried and cache=none gave the best performance
    – FrontSlash
    Dec 3, 2013 at 15:36
  • That's because people use KVM on distributions that don't do proper testing and integration, like Debian.
    – dyasny
    Dec 3, 2013 at 16:16
  • @FrontSlash So the "do not use caching" is propably your best path here - if you do not want to abandon KVM and/or Debian.
    – Nils
    Dec 4, 2013 at 14:27
  • @dyasny are there other distributions that do proper testing and integration?
    – Nils
    Dec 4, 2013 at 14:28
  • @Nils, your link to "do not use caching" is broken, was it anything other than setting cache=none for the guest?
    – FrontSlash
    Dec 4, 2013 at 15:03
-1

Don't use cache with your kvm guest, if the guest give a write request and request goes to the host cache(cache data are in memory of the physical host) and your host crash and the request are not written to the guest virtual disk, maybe you will get a filesystem problem on your guest

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .