Do Hyper-V VM snapshots grow continuously as writes are made to the HDD?

Question

(Note that while this question is about Hyper-v specifically, I'm actually interested in a generalized VM snapshot answer, unless the specific answer for Hyper-v does not apply to such a general explanation.)

I work at a large company with decent-sized VM infrastructure (a few thousand VMs). One of my server engineers is telling me that they do not allow the saving of VM snapshots for very long - they will allow a snapshot to be taken as a fallback before making a significant change to a VM, but that they need to delete it shortly afterwards (a few days or so, once we're sure our changes did not break anything).

I'm fine with that procedure - I don't expect snapshots to serve as a proxy for actual backups etc. And I can respect their desire to conserve space in the environment. What I disagree with is his reasoning. He says that the reason they need to delete them afterwards is that "snapshots can grow unbounded, every time you write to the HDD, it writes additional data to the snapshot, without limit. This is different from when you provision the original virtual HDD, where you can specify a maximum size. You cannot specify a maximum size for a snapshot."

From my understanding, snapshot images are a DELTA from the parent disk image. E.g., if I have a block on the original image that looks like:

0101 0101 0101

... and I then rewrite the middle section like so:

0101 1111 0101

... then the snapshot only stores the DIFFERENCE between the two (plus some data structure overhead that I'm sure adds complexity, but not significant from a storage perspective). Further, I understand that if I were to go rewrite those blocks back to their original state, the delta would then discard that block (so that future reads for that block would read through to the original image).

(I don't know much about HOW the snapshot stores the difference - I'm sure there are very complex structures which are necessary to keep it all organized. I'm only interested in the principle that it DOES store the difference, but not a "running history" of changes.)

He says that snapshots do not work like that - he says that if I have a block of data, I change it, and then I change it back, that EVERY time I do this, the snapshot will grow, eventually eating up a lot of disk space.

It was my understanding that a snapshot could never exceed the size of the original image (e.g. if you flipped literally every single bit on the HDD, the delta would store that), with maybe some constant overhead size as well. He seems to think that this is not true, that a VM snapshot will grow unbounded as more and more writes are made to the virtual HDD.

Am I misundersanding something about how VM snapshots work?

Answer 1

Your engineers are following a good practice, but for the wrong reasons. You are correct in that the VHDX (or whatever virtual disk technology is being used) will:

• Re-use written blocks on rewrite rather write all-new
• Have a hard size limit equal to the maximum configured size for the parent virtual disk. The reason that you cannot specify a maximum size for a snapshot is because the parent VHDX has already specified it.

However, I am unaware of any mechanism that will discard previously-written deltas if the block is returned to its original state. The performance overhead of running a difference algorithm on source and delta blocks vs. keeping a simple record of block writes would be substantial even at a relatively small scale.

Unless the VM has a great deal of disk churn, you're probably not going to see its snapshot grow horribly, though.

A VM with a single snapshot also does not have any significant performance loss, although I don't see that mentioned anywhere.

Snapshots have three very real problems:

• Environmental problems can result in orphaned AVHDX disks
• Every minute that the snapshot exists, it moves along the spectrum from "valuable" toward "liability"
• Data is not duplicated

Also, even though a snapshot cannot grow truly unbounded per se, imagine an environment with no controls on snapshots. A single snapshot could theoretically grow to double its parent's allocated size. I believe that Microsoft instituted a hard cap of 50 snapshots per VM, but only as a "OK now you're just being silly" sort of fail-safe, not because the technology requires it. So, the theoretical upper bound for a VM is 51x allocated size. While that's not a thing that's likely to ever happen, you can see how even having a few VMs with multiple snapshots could give your storage admins a headache. It certainly serves in favor of instituting a reasonable snapshot usage limitation.

Environmental Problems for Snapshots

A lot of things could fit as root cause for problems in this category. They all boil down to one fundamental problem: if the parent VHDX is modified in any way, then the AVHDX is completely invalidated and utterly useless. If the owning VM is powered on, then such modifications should be prohibitively difficult. But, if the owning VM is off, then the parent VHDX is just a file. Hyper-V or your other systems won't know anything is wrong until you try to access the child AVHDX.

The longer the snapshot exists, the higher the likelihood that something will go wrong, especially in an environment with multiple administrators. If a VM has multiple snapshots, then the problems potentially compound.

This problem is not particular to snapshots; these problems can occur with any virtual disk differencing system.

Snapshots Devalue with Age

This is really the primary reason to not keep snapshots around for long. As you correctly surmised, the differencing mechanism does not keep a historical record of changes; only the most recent change to a block is kept. So, you have only the virtual machine as it exists now in post-snapshot form, and the VM as it existed when the snapshot was taken. You can either revert to the old or keep the new. There is no in-between.

For argument's sake (and because this has happened), let's say that you have a small Exchange environment that all runs off of a single virtual machine. You take a snapshot prior to upgrading from Exchange 2013 to Exchange 2016. And then you let it run for a year. What good is that snapshot? Would you revert to it? Care to guess how long that merge will take when you delete it?

Snapshots Do Not Duplicate Data

The purpose of a snapshot is to quickly snap a virtual machine back to a point in time. It does so by directly modifying the state of the virtual machine. At no time does it make a duplicate of the data. If the AVHDX is damaged, then only the parent holds valid information and any changes made since the snapshot are lost. If the parent VHDX is damaged, then both files are useless. Also, I'm unaware of any tools that can dip into an AVHDX and pull out only changed data. Therefore, for maintaining different states over a meaningful period of time, backup is your best option. It's not as fast or convenient to work with as a snapshot, but it addresses every other problem.