2

I am running 30 VMs on RAID 0 SSDs.

The VM workloads are a heavy Docker environment (docker-compose to be precise, running around ~40 containers).

CPU load average and RAM usage are both well within comfortable parameters on the server. Disk Utilization however is massive, and iostat shows that the TPS and MB_wrtn figures are where the work is being done:

Device:  tps       MB_read/s   MB_wrtn/s   MB_read   MB_wrtn
md5      2825.57   3.28        28.15       1673116   14379843

Currently I'm defining my VM disks as follows:

<driver name='qemu' type='raw' cache='none' io='threads'/>
<source file='os.img' aio='native'/>

My VM host is using kernel 3.10.0-1062.18.1.el7.x86_64 on CentOS Linux 7 (Core) and, as a result, the deadline scheduler. The guests are using a much newer kernel and have defaulted to the mq-deadline scheduler.

I'm struggling to find any real information about optimisation, and lots of conflicting advice about which caching/io strategies to use.

This is really difficult to benchmark - the heaviest part of the workload can take 2-3 hours to kick in, and it's only for about a ~30 minute period that disk utilization gets hammered - but this is a crucial part of the work, and it is causing major slowness in comparison to when the disk utilisation is lower.

My questions are therefore:

  • What combination of cache, io and aio will offer the best performance for a high TPS/write workload?
  • Should I use iothreads given that I have "spare" CPU resource?

In addition, specifically related to my host's kernel version:

  • Should I upgrade to kernel version 3.17+ to access blk-mq (block multi-queue)?
  • If so, how do I enable this in my QEMU/KVM setup/definitions?
  • What's the best guest scheduler to use with blk-mq - is it just none?

I will be awarding a bounty for this question as soon as I am able.

=== EDIT ===

  • I have updated the iostat output above after a full, usual workload
  • We're using 4x 2TB Samsung PM883 SSDs in a RAID0 array (software RAID)
  • Added some benchmarking stats below:

fio / ioping from the host's RAID0 array

Jobs: 1 (f=1): [m(1)][100.0%][r=421MiB/s,w=139MiB/s][r=108k,w=35.7k IOPS][eta 00m:00s]
---
9 requests completed in 1.65 ms, 36 KiB read, 5.45 k iops, 21.3 MiB/s

fio / ioping from the host's RAID1 OS array

Jobs: 1 (f=1): [m(1)][100.0%][r=263MiB/s,w=86.7MiB/s][r=67.3k,w=22.2k IOPS][eta 00m:00s]
---
9 requests completed in 1.50 ms, 36 KiB read, 6.00 k iops, 23.5 MiB/s

fio from a VM's vda device

Jobs: 1 (f=1): [m(1)] [100.0% done] [246.2MB/82458KB/0KB /s] [62.1K/20.7K/0 iops] [eta 00m:00s]

I have tried tuning each VM to a maximum of 2300 read/800 write iops - but, for some bizarre reason, this makes things significantly worse. I encounter a much higher number of timeouts and job failures.

Here's what Grafana looks like during the middle of a workload:

grafana dashboard

2
  • I have no profound insight or experience with qemu/KVM but your throughput numbers don't look that high (1-12 MBps). I'm not sure what kind of disk storage are you using but for SSDs the tps value 852 doesn't have to be a problem either. May 7, 2020 at 10:51
  • Thank you @JurajMartinka. I've updated the OP to include some more accurate stats after a full workload - the SSDs are hitting about ~2500 tps. You can see from Grafana that generally when the utilization hits 100%, the iops die - I suspect this is the underlying problem.
    – turbonerd
    May 7, 2020 at 13:21

1 Answer 1

2

Probably the root of the problem is that you have SSDs with a SATA interface. SATA has only one command queue, i.e. multiple commands on the SATA bus cannot be in flight in parallel. SAS improves this a bit, but it really got massively better with NVMe, which has up to 64k independent queues.

Increasing the queue depth in SATA increases the latency of single operations (while increasing the thoughput). This is probably what happened when you increased VM IOPS.

I wouldn't expect a huge gain for a SATA SSD with blk-mq. Same goes for more iothreads. They can't work around contention on the SATA interface.

Although this is probably not what you are looking for, I guess your best option is a hardware upgrade to NVMe based SSDs. Scheduler tuning cannot work around hardware limitations.

2
  • Thanks Bernhard. Do you have any references for this? I'd love to see a comparison of SATA SSD vs. PCIE NVME specifically with regard to queue performance. How does RAID0 figure in to this equation? Would RAID0 give me better capabilities, or worse? Would it be more sensible to split my 4 RAID 0 disks out into separate disks and mount some VMs on each disk?
    – turbonerd
    May 8, 2020 at 9:01
  • gentle nudge... :)
    – turbonerd
    May 12, 2020 at 10:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .