I currently have large KVM nodes utilizing between 8 & 16 RAID10 arrays & hardware raid. We typically provision a single large volume (VM's are backed up off site as well).
KVM VM's using LVM volumes
Currently, on our 16 disk arrays, we're getting between 500 MB/s to 1.3G/s sequential write speeds using dd & 512-2048 file size at the VM level.Host level is consistent 1.4G/s which calculating the write speeds of the actual disks looks to be maxing out the disks themselves.
Hardware raid cards have onboard 2GB ram for caching.
To clarify, there's not any performance issues in terms of disk I/O (nearly no io wait with about 15-20 VMs)
We're exploring the addition of trying to add PCIe SSD cards for caching, but would ideally like to be able to implement it to new systems as well as current systems.
We could go the LSI & cachecade route - no brainer there. We could also do all SSD, no brainer there however, we'd like to implement in addition to the large sata arrays and ideally would like to use PCIe since it wouldn't require additional bays.
Any pointers on how to do this? Doesn't seem to be much info out there and many vendor websites are horrible with describing how their products actually work.