slow disk writes between host and guest

Question

I've got a ubuntu (server kernel) on a amd x4, 4gb ram, 2x seagate sata 1 tb disks for testing virtual machines and the write performance is very slow. The two disks are in a software raid1 array, one small boot ext3 partition, 10gb system partition and the rest is a xfs partition (about 980) gb for data (virtual machines).

If I'm copying files from the virtual machine to the host with rsync or scp the copy frequently stalls or goes at about 1mb/s. What's wrong?

I've tried disabling barriers on xfs, increased logbufs, allocsize, but it seems nothing helps.

The strange thing is that await (for example during copying) for sda is usually under 100, while for sdb is around 400.

Any ideas on what could be wrong / what could I do to improve this setup?

Answer 1

Do you get similar performance issues rsyncing the files from one location on the host filesystem to another location on the same array? If so then I suspect you are just seeing the latency of drive head movements when reading from one part of the drive and writing to another part of the same drive (or drives in this case, as any write needs to go to both in the RAID1 array). RAID1's write performance is essentially the same as that of the slowest single drive in the array. I wouldn't expect it to be as slow as ~1Mbyte/s though with modern drives, unless you are copying lots of small files (in which case there is much latency added for each file as directory entries are created/updated so such a low speed might not be surprising).

Your choice of filesystems and related options in the host and VMs can make a difference too. If you have journalling turned on in the VM that will increase the overhead especially if it is full journalling and not just meta-data journalling. I'm not sure about XFS, but ext3 defaults to meta-data journalling (the journalling options generally being none, meta, and full). If you have full journalling on the filesystem in the VM and on the filesystem the VM's vdisk files are stored in on the host then each write operation in the VM will become at least four physical write operations ("write to journal" and "write to main store" in the VM both of which operations become "write to journal" and "write to main store area" in the host OS's filesystem).

If this is a test machine and you don't care about the VMs dying due to disk failure because you can recreate them easily (or have them fully backed up elsewhere) then you will find RAID0 better for performance, though obviously the usual RAID0 risk applies (if you lose one drive, the whole array is dead). You could also just have the two drives as separate volumes - spreading VMs (and tasks in the host OS) over distinct spindles intelligently can give a good performance boost as VMs (and host tasks) will compete less with each other for I/O operation bandwidth. You could also use four drives and go RAID10 (which essentially gets you the benefits of both RAID0 and RAID1 in exchange for a little extra complication and more drives). Before thinking about rearranging your disk sub-system though, make sure that the performance issue you are currently seeing is not due to something else like one of the current drives developing a problem. Have you checked the SMART logs on the drives for any warning signs?