I have a server running Xen and a couple of VMs. I'm trying to set up a RAID array that's dedicated to one of the VMs in particular, which will be used for various storage-intensive purposes. Currently, I'm having a very odd performance drop when writing from the domU, which is a Debian PV guest with plenty of vcpus and memory.
Right now, the setup is that I have three 3TB WD Red hard drives arranged in a (software) RAID 5 array on the dom0. Currently, the dom0 is exposing the
/dev/mdX block as
/dev/xvdb on the virtual machine (the VM has
/dev/xvda from a LVM volume). The relevant bit of the xen config:
disk = [ <LVM stuff for xvda>, 'phy:/dev/md0,xvdb,w' ]
/dev/md0 has a ext4 filesystem on it, with the
noatime, nodiratime options in use. When I do a speed test on the filesystem from the domU, I get something like this:
# dd if=/dev/zero of=a_file bs=1M count=1024 conv=fsync 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 16.2694 s, 66.0 MB/s # dd if=some_other_noncached_file of=/dev/null bs=1M 4096+0 records in 4096+0 records out 4294967296 bytes (4.3 GB) copied, 26.379 s, 163 MB/s
However, in dom0, I get:
# dd if=/dev/zero of=a_file_somewhere bs=1M count=1024 conv=fsync 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB) copied, 6.80677 s, 158 MB/s
... which, clearly, is a very large speed difference. While 66 MB/s should be good enough for now, I'd really appreciate it if anyone can come up with an explanation for why I'm losing 60% of my write performance? I'd be expectant of 10% or so, but not 60%.
It's not dom0 resource starvation, because I've been giving it significantly more resources than it should require and the problem still occurs. It's not domU resource starvation, either, and I've pinned CPUs to stay on the same NUMA node, and the same thing results.
Here's some maybe-relevant things from
host : <hostname> release : 3.16.0-4-amd64 version : #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) machine : x86_64 nr_cpus : 24 max_cpu_id : 63 nr_nodes : 2 cores_per_socket : 6 threads_per_core : 2 cpu_mhz : 2660 hw_caps : bfebfbff:2c100800:00000000:00003f00:029ee3ff:00000000:00000001:00000000 virt_caps : hvm total_memory : 24573 free_memory : 12011 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 4 xen_extra : .1 xen_version : 4.4.1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : xen_commandline : placeholder dom0_mem=8192M dom0_max_vcpus=8 dom0_vcpus_pin cc_compiler : gcc (Debian 4.9.2-10) 4.9.2 cc_compile_by : ultrotter cc_compile_domain : debian.org cc_compile_date : Thu Jun 11 18:24:17 EEST 2015 xend_config_format : 4
If there's anything else that might be useful, please let me know and I'll update the question.