2

Is there any existing system for efficiently tracking the version history of a virtual disk image?

Essentially, I have a disk image which is used as a "template" root filesystem for testing programs. I occasionally need to make changes to it (e.g. installing packages), but most of the time it is read-only. I need to be able to keep the full history of changes (mainly so that I can roll back if something goes wrong, but I also may need to look up an older revision by date & time in order to reproduce an earlier test.)

I'd like the history to be stored in some sort of incremental format, since most of the time the changes I'm making are very small (and the complete disk image is huge.)

I'd like it if each revision were stored in a separate file, for ease of backing up with rsync.

I'd like to be able to easily delete old revisions that are no longer needed.

I'd like it if the most recent revision were stored as something close to a flat file, i.e. the read performance should O(1), independent of how many older revisions there are.

Oh, and it needs to be possible for a VM to safely use the disk image (in read-only fashion) while I am making modifications; "committing" has to be an atomic operation.

(Obviously I expect there to be some tradeoffs among these criteria; I'm just trying to give an idea of what I'm looking for.)

Performance for the actual "version control" operations is comparatively unimportant; correctness and stability, on the other hand, are crucial.

I am currently using kvm / qemu, with qcow2 disk images, but I'd be interested in hearing about other options.

I can see how I might write such a version control system myself, using qemu-img with backing files and rebasing, but is there any existing set of tools that would work for this purpose?

1
1

It sounds like VAGRANT (https://www.vagrantup.com/) could be a good match for your needs. Vagrant works with several VM engines (e.g. virtualbox, vmware, check link above).

Some advantages include:

  • It's compatible with VCS systems (single text file)
  • Cross platform - windows, linux, mac
  • Extremely easy to share with anyone, anywhere
  • Provides consistent test environment

Getting going

First install vagrant and, say, virtualbox.

Then, you describe your virtual machine in a single VagrantFile which you can generate using:

vagrant init ubuntu/trusty64 [1]

Then you will have a descriptor file for your preferred OS. If you like, edit a few ip addresses, ports, packages and shared folders in the VagrantFile.

When you are done run:

vagrant up

and you have a virtual machine which you can log into using:

vagrant ssh

Customisation

If thats not enough and you have very specific requirements beyond the capabilities of vagrant, there are plenty of vagrant VMs pre-configured with configuration management systems such as cfengine, or chef, or puppet or whichever you prefer.

[1] There are lots of pre-configured vagrant vms out there, take a look at (https://vagrantcloud.com/discover/featured).

1
  • And where exactly is version control in this answer?
    – U. Windl
    Jan 14 at 6:39
1

If you are running amd64 linux images only, and would be happy with containers instead of full virtualization, maybe Docker (http://docker.io) will be a possible option.

0

Check this out:

Virtualbox: Multi-attach disk
http://virtbjorn.blogspot.co.uk/2012/12/virtualbox-multi-attach-disk.html

What happens when you assign a master disk to a VM, is that the VM creates a differencing disk, where all writes go. The master disk will be used for reading, for whatever is not overwritten on the differencing disk. Over time, the differencing disk, that is local to every VM, will grow, and a large number of the original files will be overwritten by updates. This is not essential, the concept is not about saving disk space.

See link for more info.

0

It may depend on how your VM images are stored, but using qcow2 it seems you have them as files on a filesystem. When using the OCFS2 cluster filesystem you can use reflink snapshots to "copy" any state of an image.

The main advantage is that such snapshots will do COW (Copy On Write), so that is the initial snapshot occupies "zero" (i.e.: only a few blocks) space; only when a block changes, a new block is allocated and "un-shared" with the original image.

However the disadvantages are:

  • The space needed on the underlying filesystem grows while a snapshotted image is being written to. At the extreme there will be zero shared blocks and the image needs twice the space.

  • If the original image is completely unfragmented, it will fragment after a snapshot had been created and the image is being written to. That is due to COW, and a new block "outside" the image had to be allocated, and it cannot be predicted what the next block being written to will be. When using pure SSD-based disk systems that may not be a problem, but for spinning magnetic disks this may impact performance negatively.

  • I/O to the image will be delayed while a snapshot is being made (maybe even when one is deleted).

So when "reverting" to a snapshot you can remove the bad image and reflink it again from the desired snapshot. That is very fast. Then again the new image will start out with 100% shared blocks. Alternatively you can make a plain copy of the snapshot, so you have zero shared blocks, but it needs more space (naturally) and it will take longer to make the copy.

As I'm actually using this to make automatic periodic snapshots (like hourly, daily, weekly, yearly), I know that (for an SSD-based SAN storage connected via Fibre Channel) it takes less than three seconds to make a snapshot typically, but it can also take something like 30 seconds occasionally. While taking a snapshot the VM's I/O is delayed until the snapshot has finished.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.