Maximizing throughput of virtual file systems

Question

I've been using a QEMU system to build software. The problem I've been having is that the system seems to suffer from extremely slow disk access. This is not necessarily a problem as it's not really a time-intensive operation, but I'd like to have it work as fast as possible.

Here's all I've done to attempt to speed up I/O:

• KVM is enabled. This is the difference between night and day as far as emulated processor speed is concerned.

• Qemu is run with -display none mode through a forwarded SSH port, so there's no emulated SDL display eating CPU cycles.

• The virtual hard drives are mounted with -o noatime,nodiratime to cut down on unnecessary writes.

• The hard disk images (8 and 12 gigs respectively) are in qcow2 format and were created with -o cluster_size=2M (this is the maximum–default is 65,536 bytes). Quoth the man page: "Smaller cluster sizes can improve the image file size whereas larger cluster sizes generally provide better performance."

• The filesystem is EXT4 with the -o extents option, which the man page claims "is a much more efficient encoding which speeds up filesystem access, especially for large files."

• The virtual disk images do not have any snapshots stored in them. I am unsure whether this would slow them down, but it seems like it would; the actual file image size does not grow when a snapshot is recorded, so my guess is that the data is stored in some convoluted way that makes it so the snapshot and the "working" drive will share a cluster until it is changed, then the entire 2 MB cluster is copied into new space, then the data is changed. Ingenious but totally inefficient.

My question is whether there is anything I'm missing to increase hard disk efficiency, or what has worked for people in the past in order to achieve near-native I/O speeds.

---

Here are a couple of other things that I have observed:

• When I mount the hard drive in the virtual host with -o sync the system slows way down. This seems counter-intuitive to me since the writes to the virtual hard drive, which is just a file on my actual hard drive, are already buffered in RAM before written. This is essentially double buffering to the disk, using twice as much RAM as it has to. The dramatic speedup is especially strange since doing a grep ^Dirty /proc/meminfo never says anything over 1 MB.

• The virtual machine seems to refuse to use its swap unless I set the "swappiness" (/proc/sys/vm/swappiness) to 90 or higher. While I agree this is probably a good thing in terms of disk I/O, it makes the build processes "fight" with the fs buffering for use of the RAM. Yes, I'm sure it has enough RAM.

Answer 1

Some performance tips are here, you should definitely try to use raw format instead of qcow2 and raw devices instead of image files. You should also install paravirtualized disk drivers on guests and switch disk type from IDE to virtio.