My VM templates are <4GB in size, and when I finish cloning them into a Virtual Machine, they are still <4GB in size:

The template: enter image description here

The VM created from that template: enter image description here

My ESXi machine is attached to a NAS with a 1Gbps link, the vCenter machine is running on bare metal, also attached via 1Gbps to the ESXi and NAS. If I initiate cloning the template to a VM and pop onto the NAS to check its network usage with vnstat -l, I see the 1Gbps pipe being fully used:

root@nas:~# vnstat -l
Monitoring eth0...    (press CTRL-C to stop)

   rx:     5.71 Mbit/s  7085 p/s          tx:   958.14 Mbit/s 81384 p/s

Despite this, cloning takes around 6-8 minutes:

enter image description here

Without doing too heavy of math, it would appear that this should be significantly shorter. Like < 1 minute.

What could I be doing better to speed up this provisioning time?

  • Is there a reason you are cloning the template instead of deploying a VM from the template? – D34DM347 Jan 26 '17 at 12:34
  • Sorry, that's what I do. I click "Deploy a VM from the template", it gets labeled as "Clone Virtual Machine", and it's the same API with PyVmomi, so I just call it cloning since that's apparently how VMware internally refers to it. – omghai2u Jan 26 '17 at 13:12
  • In case you are interested - I worked the math on this one, and your transfer should theoretically take around 23 seconds. However, is there any sort of encryption going on between the server and the NAS? I know that VMware Converter is notorious for taking far longer than it theoretically should, primarily due to the way that SSL encryption is set up. – Kernel Stearns Jan 26 '17 at 15:33
  • Yeah, that 23 seconds is why I'm asking the question here. There's no additional encryption. And as I show on the vnstat -l we're using 1Gbps for the entire 6-8 minutes, so I can't imagine what all that overhead is. I was hoing someone could help me understand what's going on. No, VMWare Converter is running, this is all happening from the vCenter web interface. The template was made on the same machine it's just deploying a template of that VM, so no weird hardware conversion or anything. – omghai2u Jan 26 '17 at 16:15
  • I understand you are not running Converter, but I thought maybe ESXi did a similar sort of SSL encryption. Also, just to clarify, is the ESXi server the only device connected to the NAS? If there are other servers using it, then that 1Gbps will be split up between each of them. – Kernel Stearns Jan 27 '17 at 16:09

As far as I can tell, (and given the lack of responses, I would say this is true), the transfer is taking so long because ESXi performs SSL encryption on all its transfers. I know that this is the reason that VMware vCenter Converter takes so long to perform transfers, and it makes sense that VMware would apply the same encryption to template transfers across the network. Therefore, unless there is a way to disable SSL encryption on ESXi, it is not possible to speed up the transfer time as long as it is between ESXi and the NAS.

That being said, there may be an interesting solution to your problem that would speed up the provisioning time. If you have a little extra space locally on your ESXi server, you can create a small virtual machine, transfer the template file into that virtual machine, and then deploy virtual machines from that local template instead of deploying them from the NAS. Essentially, you are creating a virtual local NAS specifically for templates.

While this sounds like a pretty complicated solution, I know from experience that it speeds up vCenter Converter by at least a factor of 10. I can't explain why, but I'm not going to question a 10x speed increase!

  • Someone should write a script for this using vSphere API. – mzhaase Feb 1 '17 at 15:46
  • I have tried this solution surrounding using a local datastore but see the same speed of 6-8 minutes. SSL encryption does not make sense to create such an overhead, as it's simply a bit of up-front cost, but there are relatively few number of created network streams; not enough to account for all of the additional transferring. – omghai2u Feb 1 '17 at 22:20
  • @omghai2u - That is interesting. However, what is your disk usage during the provisioning? While I can't explain why the NAS is showing a transfer speed of 1Gbps, it seems that the disk speed on the ESXi server must be slower than that. – Kernel Stearns Feb 2 '17 at 15:24
  • @KernelStearns Disk usage where? On the target ESXi host or on the NAS? And how is best to check that for what you would like to see? – omghai2u Feb 3 '17 at 13:54
  • @omghai2u Sorry I should have clarified that. I am looking for the disk usage as shown by ESXi in Monitor > Disk Usage, while you run the transfer from the local virtual NAS you created. Based on your 6-8 minute transfer time, your read/write speed should be about 80Mbps. If this is your normal maximum speed, then your transfer speed is being limited by the disk speed on the server. – Kernel Stearns Feb 3 '17 at 16:15

Sorry I didn't see this one when you logged it the other day, it's actually quite simple.

There's really only two modes for cloning - clone-by-host and clone-by-array.

Clone-by-host is when the host reads the template blocks and writes them back out to the new VM files. The time taken for this is heavily reliant on the speed of the network between host and array - in this case it's quite slow at 1Gbps and iSCSI adds its own overhead, obviously moving to 10/25/40Gbps would significantly increase this speed - though of course you could be limited at that level by the actual array size.

Clone-by-array is kind of simpler, the host sends some commands to the array to simply close a list of blocks, the array then gets on with the job itself. This is obviously a lot faster and reduces the impact of slower links between host and array BUT it is utterly reliant on the array supporting this kind of load. Specifically the array needs to support VMware's 'VAAI' protocols to do this. You can check for this support on VMware's Hardware Compatibility website but if your array doesn't support it then you're out of luck sorry, well unless a newer version of the software it runs suddenly starts supporting it anyway.

I hope this helps.

  • Does nobody who downvoted this care to comment why? He wants to know about how to speed up cloning, this is important information. – Chopper3 Feb 2 '17 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.