Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm using partclone to take disk images with this command:

partclone.ntfs -c -s /dev/$mypartition | lzop -c > /my/nfs/share/mypartition.lzo

The partition size is 34.1 GB with 33.8 GB used.

The problem is the performance drops after a while. At first the backup is being written at roughly 300 MB/minute, but the performance gradually drops to 150-200 MB/minute. The estimated time of completion increases from about 1.25 hours to about 2.5 hours.

I took an image with no compression:

partclone.ntfs -c -s /dev/$mypartition -o /my/nfs/share/mypartition

The backup speed hovers around 600 MB/minute, but the performance DOESN'T degrade after a while. It stays around 600 MB/minute more or less.

Is there anything I can do keep the performance from degrading when using compression?

EDIT: I've used gzip and lzop and both suffer from degrading performance. I also tried backing up the same image on a Pentium D machine (2 cores, 3 GB of RAM), and a Xen DomU (1 processor, 512 MB of RAM). They both have the same problem.

share|improve this question

3 Answers 3

up vote 0 down vote accepted

I would highly recommend testing this with a buffer command in between to allow the compression to slow down or speed up without slowing down the backup itself.

I highly recommend bfr, I use it in all such situations myself.

Just run partclone.ntfs -c -s /dev/$mypartition | bfr | lzop -c > whatever.dat

share|improve this answer

With the old CPUs you're using, it seems most likely that you're CPU bound; compression eats a lot of CPU (and it can vary depending on the data being compressed) and is almost certainly the immediate cause of the issue.

If you can compress with a faster CPU somehow, perhaps by piping the data to a faster machine first, then that should help.

You can also try using a multi-core capable compression utility such as pbzip2, which should increase the rate of compression on your dual-core system, at least. It won't do anything for you on the single-CPU VM, though. pbzip2 writes bzip2-compatible files and is already packaged for most Linux distributions.

share|improve this answer

Since the process is single threaded, the compression will be the bottleneck at the max speed of the CPU.

If you can break this down so that you could cut it into several concurrent portions you might be able to get a little more speed by using more of the CPU cores, if you have multiple cores (not HT).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.