Slow writes for VM in Xen?

Question

I have a server running Xen and a couple of VMs. I'm trying to set up a RAID array that's dedicated to one of the VMs in particular, which will be used for various storage-intensive purposes. Currently, I'm having a very odd performance drop when writing from the domU, which is a Debian PV guest with plenty of vcpus and memory.

Right now, the setup is that I have three 3TB WD Red hard drives arranged in a (software) RAID 5 array on the dom0. Currently, the dom0 is exposing the /dev/mdX block as /dev/xvdb on the virtual machine (the VM has /dev/xvda from a LVM volume). The relevant bit of the xen config:

disk = [ <LVM stuff for xvda>, 'phy:/dev/md0,xvdb,w' ]

The /dev/md0 has a ext4 filesystem on it, with the noatime, nodiratime options in use. When I do a speed test on the filesystem from the domU, I get something like this:

# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 16.2694 s, 66.0 MB/s
# dd if=some_other_noncached_file of=/dev/null bs=1M
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 26.379 s, 163 MB/s

However, in dom0, I get:

# dd if=/dev/zero of=a_file_somewhere bs=1M count=1024 conv=fsync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 6.80677 s, 158 MB/s

... which, clearly, is a very large speed difference. While 66 MB/s should be good enough for now, I'd really appreciate it if anyone can come up with an explanation for why I'm losing 60% of my write performance? I'd be expectant of 10% or so, but not 60%.

It's not dom0 resource starvation, because I've been giving it significantly more resources than it should require and the problem still occurs. It's not domU resource starvation, either, and I've pinned CPUs to stay on the same NUMA node, and the same thing results.

Here's some maybe-relevant things from xl info:

host                   : <hostname>
release                : 3.16.0-4-amd64
version                : #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04)
machine                : x86_64
nr_cpus                : 24
max_cpu_id             : 63
nr_nodes               : 2
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2660
hw_caps                : bfebfbff:2c100800:00000000:00003f00:029ee3ff:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 24573
free_memory            : 12011
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
xen_major              : 4
xen_minor              : 4
xen_extra              : .1
xen_version            : 4.4.1
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : 
xen_commandline        : placeholder dom0_mem=8192M dom0_max_vcpus=8 dom0_vcpus_pin
cc_compiler            : gcc (Debian 4.9.2-10) 4.9.2
cc_compile_by          : ultrotter
cc_compile_domain      : debian.org
cc_compile_date        : Thu Jun 11 18:24:17 EEST 2015
xend_config_format     : 4

If there's anything else that might be useful, please let me know and I'll update the question.

Thanks!

Answer 1

I've seen near native disk performance with PV guests, with the MD work being done at both the dom0 and domU levels.

If you can, try setting up guest as paravirtualized (PV) instead of a hardware virtual machine (HVM).

Also, for either PV or HVM, try assigning the whole drives to the guest and having it handle the RAID stuff (MD).

See what works best for you.

Answer 2

Some more investigation turned up the fact that this is not a unique issue (see for example https://bugzilla.redhat.com/show_bug.cgi?id=500145). Following some suggestions there, further investigation with iostat showed that the average write size on the domU was somewhere lower than 128KiB, while on the dom0 was about 512KiB (found by dividing wkB/s / w/s).

Some Googling turned up the fact that this is a known issue, with a patch submitted back in December to "fix" the problem by raising the default value of the kernel module parameter xen_blkfront.max from 32 to 128. By overriding the default value myself (in the GRUB configuration for the VM), the write speed on the domU jumped from about 65MiB/s to about 115MiB/s, which is enough of a bonus that I'm not going to look too much further.

So while there's still more territory to cover (from 115 MiB/s to the dom0 speed of 150 MiB/s), the major annoying part has been fixed, so I'm marking this as the solution (once I can). I'll update with any more tweaks or whatnot that I find which provide significant impact.