Thanks to Scott Pack's wonderful answer (I didn't know how to do this with ssh before), I can offer this improvement (if bash
is your shell). This will add parallel compression, a progress indicator and check integrity across the network link:
tar c file_list |
tee >(sha512sum >&2) |
pv -prab |
pigz -9 |
ssh [user@]remote_host '
gunzip |
tee >(sha512sum >&2) |
tar xC /directory/to/extract/to
'
pv
is a nice progress viewer program for your pipe and pigz
is a parallel gzip program that uses as many threads as your CPU has by default (I believe up to 8 max). You can tune the compression level to better fit the ratio of CPU to network bandwith and swap it out with pxz -9e
and pxz -d
if you have much more CPU than bandwidth. You only have to verify that the two sums match upon completion.
This option is useful for very large amounts of data as well as high latency networks, but not very helpful if the link is unstable and drops. In those cases, rsync is probably the best choice as it can resume.
Sample output:
6c1fe5a75cc0280709a794bdfd23d7b8b655f0bbb4c320e59729c5cd952b4b1f84861b52d1eddb601259e78249d3e6618f8a1edbd20b281d6cd15f80c8593c3e - ]
176MiB [9.36MiB/s] [9.36MiB/s] [ <=> ]
6c1fe5a75cc0280709a794bdfd23d7b8b655f0bbb4c320e59729c5cd952b4b1f84861b52d1eddb601259e78249d3e6618f8a1edbd20b281d6cd15f80c8593c3e -
For block devices:
dd if=/dev/src_device bs=1024k |
tee >(sha512sum >&2) |
pv -prab |
pigz -9 |
ssh [user@]remote_host '
gunzip |
tee >(sha512sum >&2) |
dd of=/dev/src_device bs=1024k
'
Obviously, make sure they're the same size or limit with count=, skip=, seek=, etc.
When I copy filesystems this way, I'll often first dd if=/dev/zero of=/thefs/zero.dat bs=64k && sync && rm /thefs/zero.dat && umount /thefs
to zero most of the unused space, which speeds up the xfer.