-1

I know this question has been asked before and I saw the recommendations on using pigz. Good recommendations, but I wanted to see if I really have a "problem".

I have an esxi host. It's the free version so a lot of the enterprise tools don't work. It's also personal lab use so I'm not concerned greatly with high availability.

Currently to back up VMs, I have a script which shuts the VM off, copies the VM files from local storage to a 1gbe NAS, powers on the VM, and then tar -z the files over on the NAS itself.

I started recording some times just to see how long things took.

Example 1:

  • 28GB VM vmdk file
  • Time to copy: 5 min 21 sec
  • Avg speed on copy: 713mbps
  • Time to zip with compression: 1 hour 55 min
  • Avg speed on zip w/ compression: 33mbps

Example 2:

  • 95GB VM vmdk file
  • Time to copy: 29 min 7 sec
  • Avg speed on copy: 445mbps
  • Time to zip with compression: 4 hour 3 min
  • Avg speed on zip w/ compression: 53mbps

Really this is a non-issue since the VM starts up right after the copy completes, the tar -z could run for a day and not be noticed. Just wondering if that is normal speed for tar -z?

I checked the VM host and CPU and storage seem to both be idling. I checked the NAS and CPU and storage seem to be idling. I don't know if I want to go the full on pigz route and have it max out the VM host CPUs, on the other hand, a bit more speed would be nice.

Improve this question
2
  • 1
    Starting a VM with the copied image is not ideal. Mounting the copied image directly on the host system would be much faster. (Just remember to install all security updates for the file system drivers.)
    – kasperd
    Apr 16, 2018 at 21:02
  • Thanks, I'm just using this as a poor man's backup. So I'm not actually ever starting the copied VM unless I have to restore it for some reason.
    – sspeed
    Apr 17, 2018 at 18:46

1 Answer 1

Reset to default
0

You could use pigz but leverage the -p or --process option

From the man page at https://zlib.net/pigz/pigz.pdf

-p --processes n
Allow up to n processes (default is the number of online processors)

This would allow for controlling the amount of cpu impact the compression had on the rest of your environment.

If you had another host you could farm off the compression portion of the operation to it and allow for using all the available cpus with pigz.

As for the performance of the initial copy vs the compression task. Depending on the NAS configuration you may have some amount of contention as you are reading from and writing to the same system. Performing a simple copy from the NAS to the NAS would give you a decent estimate of how fast you could hope to copy if the CPU wasn't the bottleneck for the compression.

Improve this answer
1
  • Thank you! I did pigz with a -p 4 (I show 7 working processors for whatever reason). It dropped the compression time out on the NAS from one hour and 55 min to just 8min for a 28GB file. I saw CPU spikes on the esxi host to around 70% CPU, but all still very available.
    – sspeed
    Apr 16, 2018 at 20:44

Not the answer you're looking for? Browse other questions tagged .