5

One of our highly utilized vm is giving the error that there is no more space for the the vmdk shown in the following image.

enter image description here

The VM won't start because of this issue, so I'm not able to delete data from the volume at OS level.

The virtual machine has 3 snapshots but the consolidate option is greyed out. Screenshot from the vm's datastore belowenter image description here

Improve this question
5
  • Which ESXi version is that server running?
    – BeowulfNode42
    Nov 28, 2016 at 8:17
  • @BeowulfNode42 esxi 5.5
    – joebegborg07
    Nov 28, 2016 at 8:21
  • added to 'cautionary-tales-thinning-in-esxi-rather-than-on-array.xls' :)
    – Chopper3
    Nov 28, 2016 at 9:34
  • 1. There's no risk in deleting the log files. If you want to preserve them for some reason you can copy them from the datastore to somewhere else and then delete them from the datastore. 2. If deleting data in the guest OS volume would actually work then you can simply attach the VMDK to another like VM and delete data in the volume from within that VM.
    – joeqwerty
    Nov 28, 2016 at 10:39
  • Thanks @joeqwerty. I've tried consolidating the virtual machine, however I got the error that there's not enough disk space. Can anyone help me with what disk space is required ? Is it double to present vmdk disk ? Is it double the combined size of all vmdks ?
    – joebegborg07
    Nov 28, 2016 at 11:50

3 Answers 3

Reset to default
4

You need to either

  • Extend DataStore size (can be done online) or
  • Storage vMotion some of the other VMs off this Data Store.

The ultimate goal is to free up space on the DataStore so you can unfreeze VM. And by the way - freeing space from the OS level won't help you.
If your vmdk is thin - it will grow in size over time, but will not shrink when you clean underlying OS.
There are some tricks to reduce vmdk size again available, but those require bringing VM down (not online process).

Improve this answer
1
  • FYI Joe, from the looks of the picture there is no value written in the provisioned size column for the base disks, so they should be thick provisioned disks. As far as I know snapshots are always thin provisioned, but only the current snapshot file will be growing.
    – BeowulfNode42
    Nov 28, 2016 at 8:47
4

While I've not run in to that particular problem before you might try

  • Free up some space and try again, perhaps enough for the consolidate to work by
    • Delete some old log files. I see that you have over 2GB in logs in that one folder alone.
    • You may find that shutting down some VMs will free up some space taken by the *.vswp files, while they are shut down.
    • What Dmitry Zayats suggests about extending the datastore to make it bigger or moving other VMs off that datastore.
    • If all else fails, move the VM to a datastore with enough space, then see if you can get rid of the snapshots. Note that since the machine is not running you can use a few different methods of moving the VM.
  • Try the "Delete All" button in the snapshot manager instead of consolidate.

Note: Deleting data at the guest OS level will likely only make the snapshot files bigger, as it makes the snapshots more different to the previous snapshot than before.

Some light reading

Improve this answer
5
  • @BeofuldNode42. Is there a risk of corruption when deleting older log files ? Sorry for the simple question, I'm new to vmware.
    – joebegborg07
    Nov 28, 2016 at 8:20
  • The "delete all" option is very much like going through each snapshot and clicking delete on each one, except it does all of them in one go. Though I've never done it with a full datastore before. You need at least some free space. Also any corruption that has been introduced in to the data stored in the vmdk by the guest crashing will be present, and you will no longer have a snapshot to go back to.
    – BeowulfNode42
    Nov 28, 2016 at 8:42
  • Thanks for your info in reply. I tried consolidation and I get the 'consolidation was successful message' after 2 seconds. The vms still have the same error and the snapshots are still visible in snapshot manager. So I suspect the consolidation did not work. Why would this happen? Could it be because the data store has 18GB left of disk space?
    – joebegborg07
    Nov 28, 2016 at 9:23
  • Would vmkfstools -K help/work on a disk in that state?
    – rackandboneman
    Nov 28, 2016 at 11:50
  • not sure about the K or -k options, but possibly the vmkfstools --fix check disk.vmdk and then vmkfstools --fix repair disk.vmdk and you may want to look in to the --chainConsistent option too. I hope you have a good backup as well...
    – BeowulfNode42
    Nov 29, 2016 at 0:33
0

My way in such situations is:

  1. free up space (as described by BeowulfNode42 )
  2. increase Datastore size (as described by Dmitry Zayats )
  3. login using SSH to ESXi and use vmkfstools to consolidate disks, to make them thin (if you clone a snapshot of disk, you will get a freshly consolidated disk, but you need space)
  4. the last resort: copy vmdk files to external storage and use vmware-vdiskmanager from VMWare Workstation

The 4th way helped me in worst cases and it's most complicated way. Here are details:

  1. Shutdown problematic VM
  2. Backup it by copying all its files to external storage (configured in ESXi or using scp)
  3. Copy VMDK files into another place (sort of sandbox) for further risky operations. You should see small descriptor files mydisk.vmdk and data files mydisk-flat.vmdk
  4. Using VMWare Workstation create flat disks stored in single file. (I do it by creation of temporary new VM)
  5. Remove data files generated by VMWare Workstation and descriptor files from ESXi
  6. Using any text editor change the descriptor files generated by VMWare to describe data files from ESXi (you will need to change size and probably cylinder count)
  7. Make your operations (consolidate disks using vmware-vdiskmanager, start VM, free space inside VM, fill free space with zeros (dd if=/dev/zero of=/zer0 && rm -f /zer0), shrink disks)
  8. On ESXi server note configuration of VM and then delete it
  9. Since you now have some free space on ESXi it's a good time to cleanup other VMs
  10. Copy updated VMDKs to ESXi
  11. Recreate deleted VM with new disks. (ESXi should detect what disks are from VMWare Workstation and should propose to convert them. Otherwise use vmkfstools for conversion)
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .