what is the fastest and most reliable way of transferring a lot of files?

Question

I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would max out a 100mbit connection over the internet and be very reliable.

Answer 1

Have you considered Sneakernet? With large data sets overnight shipping is often going to be faster and cheaper than transferring via the Internet.

Answer 2

You mention "rsync," so I assume you are using Linux:

Why don't you create a tar or tar.gz file? Network transfer time of one big file is faster than many small ones. You could even compress it if you wish...

Tar with no compression:

On the source server:

tar -cf file.tar /path/to/files/

Then on the receiving end:

cd /path/to/files/
tar -xf /path/to/file.tar

Tar with compression:

On the source server:

tar -czf file.tar.gz /path/to/files/

Then on the receiving end:

cd /path/to/files/
tar -xzf /path/to/file.tar.gz

You would simply use rsync to do the actual transfer of the (tar|tar.gz) files.

Answer 3

You can use various compression options of rsync.

-z, --compress              compress file data during the transfer
     --compress-level=NUM    explicitly set compression level
     --skip-compress=LIST    skip compressing files with suffix in LIST

compression ratio for binary files is very low, so you can skip those files using --skip-compress e.g. iso, already archived and compressed tarballs etc.

Answer 4

You could try the tar and ssh trick described here:

tar cvzf - /wwwdata | ssh root@192.168.1.201 "dd of=/backup/wwwdata.tar.gz"

this should be rewritable to the following:

tar cvzf - /wwwdata | ssh root@192.168.1.201 "tar xvf -"

You'd lose the --partial features of rsync in the process, though. If the files don't change very frequently, living with a slow initial rsync could be highly worth-while as it will go much faster in the future.

Answer 5

I'm a big fan of FTP. I use FTP to transfer media from my main computer to my server. I get good speeds, over LAN.

FTP is reliable, I'd give that a shot, as it's easy to set up, and it could be faster in some cases.