1

So I have a particularly troubling issue left behind by the old IT department. We're running several snapshots in and no one thought to consolidate them, likely because no one was skilled enough in VMWare to realize they had to do this. So, this is the problem I'm left with:

Nested Snapshots

Assuming the image doesn't load, we have this:

  • ->The VM 343.7GB
  • --> Snapshot 1 5/14/2018, 150 GB
  • ---> Snapshot 2 06/13/2018, 9.03 GB (Snapshot the VM's memory is: NO)
  • ----> Snapshot 3 06/13/2018, 31.19 GB
  • -----> Snapshot 4 06/14/2018, 386.24 MB
  • ------> Snapshot 5 08/27/2018, 45.43 GB
  • -------> You are here (yay) 59.5 GB

I did some digging and what looks to be the best fix is to do this:

  1. Create a backup of the VM files, ideally while it is powered off. Just copying the whole VM to a second location should suffice.
  2. Delete the snapshots. Ideally during non office hours, the consolidation will take time. A lot of time. It will go faster when the VM is turned off.
  3. Check if the VM is intact, if not, restore the backup. Source: VMWare old snapshot consolidation

My question to you is:

These snapshots have been running for quite some time, the oldest since 2018. Based on what I'm reading, it's likely to cause some degradation if not completely corrupt the VM at this point.

  1. Is it worth my time to attempt the above fix?
  2. If not, am I better off backing up the databases stored on the server and reverting the server?

If I understand correctly, reverting returns it to the state before the snapshots occurred, discarding the changes made in the snapshot. Whereas deleting the snapshots consolidates all of the changes made into one. (Weird terminology VMWare)

Also, this is a thin-provisioned server. Running out of disk space is what made me discover this issue, and I've got about 4GB left at this point.

  • I’d you’re out of disk space, I don’t think there’s much you can do here, as I seem to recall the consolidation/deletion prit takes some extra space. I’ve never actually monitored my disks space that closely while cleaning snapshots, so I might be wrong. Do you have any other space available elsewhere? Also, what version of vSphere? – GregL Jan 24 at 0:20
  • GregL, see my answer below, as long as you're deleting intermediary snapshots, disc space requirement shouldn't grow. – Stuggi Feb 16 at 9:56
1

According to "Estimate the time required to consolidate virtual machine snapshots (2053758)" at https://kb.vmware.com/s/article/2053758, an additional Delta file is built during consolidation if the VM is powered on. This is in the NOTES section and states

If disk consolidation is started when the virtual machine is powered on, an additional delta file is created to track the modified blocks, which is finally written to the base disk at the end of the consolidation. However, no additional delta file is required when deleting only one snapshot which is not the current one.

"No additional delta file is required when deleting only one snapshot which is not the current one." Deleting one at a time in oldest to newest order will not consume additional drive space until you get to the most recent snapshot. This would run in the background.

A VMware community thread at https://communities.vmware.com/thread/560315 also has this issue. Worst case scenario, the base/parent disk should only be able to grow up to the amount of data in the snapshots.

Additionally, here is a VMware KB about consolidating snapshots in ESX 3.5 and ESX 4.0 (which requires patch updates). ESX 5 and higher have this function built in and performs the same operation. It covers the same points as the community discussion I linked. https://kb.vmware.com/s/article/1023657.

As such, my answer to additional space requirements is "there are no additional space requirements if you turn the VM off first. Or you can delete one snapshot at a time, oldest to newest, and the freed space will allow you to delete the newest one while it is still powered on".

Reverting the snapshot to years ago has its own problems. You lose Domain Trust since the Computer Machine Password is no longer what it should be. Based on the timestamps you provide, you will also lose 1.5 years of Windows Updates and any other 3rd party application updates or manually performed updates. Registry changes that didn't come down through GPO. Settings. You lose any data that is stored elsewhere like documents or downloads folders (unless you have it backed up). This would all have to get put back.

If your concern is corruption, then shut the VM off, copy the disk files to additional storage, and attempt consolidation, as you have mentioned as an option. Or simply build a new server and migrate the database... if you have the space on a different location since you have mentioned space issues already.

0

I'm not absolutely sure about what your question is, so I'll try to answer both.

You most definitely want to consolidate that disc, as snapshots aren't meant to be "long term backups". When you create a snapshot (ignoring VVOLs for the sake of simplicity), vSphere "freezes" the VMDK-file (the file on the datastore representing the HDD of the VM) for the VMs discs, and starts writing all changes to a separate delta file. This file can only grow to be as large as the original VMDK (approximately, there might be some additional overhead). If you then take a second snapshot, your first delta-file is again "frozen", and vSphere starts a second delta file.

When you then remove a snapshot, vSphere takes the delta file, and writes all the changes back to the original VMDK. However, while it's doing this to a powered-on VM, it needs somewhere to keep writing the changes coming in from the VM, so it creates a temporary delta file to keep changes to the VMDK while it's consolidating the snapshot. When it's done, it again consolidates the much smaller temporary delta file, and usually stuns the VM for a fraction of a second to keep it's disc I/O quiet while it consolidates the temporary file.

However, if you remove an intermediate snapshot, vSphere already has another newer delta file to work with, so it can just do it's thing without affecting the VM. This can be exploited if you wan't to minimise the impact of removing a large snapshot, just take a new snapshot, remove the old one in the background, and then remove the much smaller snapshot when it's done.

Reverting works a bit different. Here you're rolling back the state of the VM, which is much less resource intensive, you just delete the delta file and restart the VM with the original VMDK (you can also snapshot the memory, which then returns the VM to a powered-on state).

So, knowing all this;

If your VM is working fine, albeit a bit slower due to the performance impact, you should do as follows:

  1. Ensure you have a good backup of the VM
  2. Power down the VM to speed things up
  3. Remove all intermediate snapshots, starting with the oldest one.
  4. Remove the last snapshot
  5. Run a consolidate discs if that option is available.

If your VM is broken, power it down and restore a copy from backup/rebuild it. Reverting to an ancient snapshot should be your last option. Snapshots are intended to be a "oh, I accidentally deleted the whole C: drive while performing some work, let me revert to my snapshot I took 30 minutes ago", not a backup.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.