How to speed up file transfer from server to server over the Internet? [duplicate]

Question

This question already has an answer here:

• How to copy a large number of files quickly between two servers 23 answers

How can I transfer a file from one server to another server with a great speed ? right now I am using FTP to transfer a file but it is taking so much time. To transfer a file of 2 GB its taking around 3 hours. Is there any other procedure to transfer a file which is faster than FTP. Server Location: One in India and another in US.

Answer 1

Is this normal for your location? If you transfer 2GB to other sites, what sort of speed do you usually see? What is the network speed at your location and at the remote location?

The transfer speed over long distances will mostly depend on the network bandwidth available between the two locations and for any hop in between. You are transferring data over the Internet which means that speeds can vary greatly from time to time. The Internet does not guarantee a minimum speed.

Your best bets are:

1. Before transferring the data, compress the data on your local server so that they are smaller in size.
2. Use rsync to compress the data on the fly. See the examples at http://en.wikipedia.org/wiki/Rsync#Examples
3. Break the data into chunks which you transfer one at a time. This won't speed up the data transfer, but it will make the transfer more fault tolerant (you won't need to restart from the beginning if the transfer fails 99% of the way through). Compression can help.
4. Another ISP or network may offer better transfer speeds. Try another ISP or network.

Answer 2

• Log on to one of the servers using SSH
• Open up file transfer from the server you're logged in on to the other server (e.g. lftp, scp, etc.)
• Send the file.

The last two are problably the same thing, but I wanted to get 3 steps out of it. It's as simple as "just send the file."

Answer 3

Instead of that why not to use rsync or rdiff command both are fast tools to transfer files.

Answer 4

When I was reading about file transfer performance for NAS devices, the speeds people were getting over SMB was typically always slower than FTP. I got the sense that whilst there might be a lot of things annoying about FTP, it's not in itself a slow protocol.

I wouldnt expect that, leaving everything else unchanged, merely changing from FTP to another protocol is going to give you the kind of speed improvement you seem to be expecting.

How do you know the problem isn't bandwidth? Can you try doing a test with scp or rsync and comparing with FTP?

Answer 5

Do you have TCP window scaling enabled at both ends? Even if you have a lot of bandwidth, connections between the U.S. and India will inherently have a huge bandwidth delay product. If window scaling is enabled and you're still seeing poor FTP performance you might try running rsync in parallel.

Also, stop using FTP. It's a horrible protocol. :)

Answer 6

SF community members may not be happy for taking you back to the stone age's of Unix, but for lower protocol overhead and a good compression, you might want to try a combination of dd + netcat + bzip2. Yes, this is not going to be secure, so you have to close the ports for all except for the two nodes. No guarantee, no security, no authentication ... but it is faster.

1 - compress your file using bzip2 to get say , file.bz2

2 - Listen using netcat on node2

     nc -l 6668 | dd of=/dir/file.bz2

1. Push it from node1

    dd if=/dir/file.bz2 | nc node2 6668
   

Answer 7

If youre trying to transfer a single 2GB file ftp is one of the most clean and fast protocols, you can also safely use resume feature in case of interruption during the transfer.

Ftp is a bad idea if you're trying to transfer a great number of little files, because you have a huge useless activity opening and closing connections for each file.

Rsync(via ssh) or scp(sftp) are good alternatives, but you must consider that encryption significantly reduce throughput. You can use a light algorithm like blowfish ('scp -c blowfish' or 'rsync -e "ssh -c blowfish"') but the speed will always lower than ftp.

Netcat is an alternative, but only on lan.

Answer 8

If your file is similar to one already at the destination you could use rsync with the --fuzzy option. Fuzzy will look for a similarly named file in the destination directory and use it as the starting point for the copy sending only the differences between the two files across the wire.

As an example, here I'm copying xwiki-enterprise-web-3.4.war into a remote directory which already contains a xwiki-enterprise-web-3.2.1.war.

$ rsync -av --fuzzy xwiki-enterprise-web-3.4.war example.com:Downloads/xwiki-enterprise-web-3.4.war
sent 69065310 bytes  received 74176 bytes  389518.23 bytes/sec
total size is 138567168  speedup is 2.00

You'll notice only about 1/2 the contents of the file actually needed transfer. It would be even better if the files were more similar as in this specific case the newer version xwiki is actually 26M larger than the older copy.

$ du -sh xwiki-enterprise-web-3.*
107M    xwiki-enterprise-web-3.2.1.war
133M    xwiki-enterprise-web-3.4.war