I am running 30 VMs on RAID 0 SSDs.
The VM workloads are a heavy Docker environment (docker-compose to be precise, running around ~40 containers).
CPU load average and RAM usage are both well within comfortable parameters on the server. Disk Utilization however is massive, and iostat
shows that the TPS and MB_wrtn figures are where the work is being done:
Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn
md5 2825.57 3.28 28.15 1673116 14379843
Currently I'm defining my VM disks as follows:
<driver name='qemu' type='raw' cache='none' io='threads'/>
<source file='os.img' aio='native'/>
My VM host is using kernel 3.10.0-1062.18.1.el7.x86_64
on CentOS Linux 7 (Core)
and, as a result, the deadline
scheduler. The guests are using a much newer kernel and have defaulted to the mq-deadline
scheduler.
I'm struggling to find any real information about optimisation, and lots of conflicting advice about which caching/io strategies to use.
This is really difficult to benchmark - the heaviest part of the workload can take 2-3 hours to kick in, and it's only for about a ~30 minute period that disk utilization gets hammered - but this is a crucial part of the work, and it is causing major slowness in comparison to when the disk utilisation is lower.
My questions are therefore:
- What combination of
cache
,io
andaio
will offer the best performance for a high TPS/write workload? - Should I use
iothreads
given that I have "spare" CPU resource?
In addition, specifically related to my host's kernel version:
- Should I upgrade to kernel version
3.17
+ to accessblk-mq
(block multi-queue)? - If so, how do I enable this in my QEMU/KVM setup/definitions?
- What's the best guest scheduler to use with
blk-mq
- is it justnone
?
I will be awarding a bounty for this question as soon as I am able.
=== EDIT ===
- I have updated the
iostat
output above after a full, usual workload - We're using 4x 2TB Samsung PM883 SSDs in a RAID0 array (software RAID)
- Added some benchmarking stats below:
fio / ioping from the host's RAID0 array
Jobs: 1 (f=1): [m(1)][100.0%][r=421MiB/s,w=139MiB/s][r=108k,w=35.7k IOPS][eta 00m:00s]
---
9 requests completed in 1.65 ms, 36 KiB read, 5.45 k iops, 21.3 MiB/s
fio / ioping from the host's RAID1 OS array
Jobs: 1 (f=1): [m(1)][100.0%][r=263MiB/s,w=86.7MiB/s][r=67.3k,w=22.2k IOPS][eta 00m:00s]
---
9 requests completed in 1.50 ms, 36 KiB read, 6.00 k iops, 23.5 MiB/s
fio from a VM's vda
device
Jobs: 1 (f=1): [m(1)] [100.0% done] [246.2MB/82458KB/0KB /s] [62.1K/20.7K/0 iops] [eta 00m:00s]
I have tried tuning each VM to a maximum of 2300 read/800 write iops - but, for some bizarre reason, this makes things significantly worse. I encounter a much higher number of timeouts and job failures.
Here's what Grafana looks like during the middle of a workload:
tps
value 852 doesn't have to be a problem either.tps
. You can see from Grafana that generally when the utilization hits 100%, the iops die - I suspect this is the underlying problem.