Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This question already has an answer here:

How can I transfer a file from one server to another server with a great speed ? right now I am using FTP to transfer a file but it is taking so much time. To transfer a file of 2 GB its taking around 3 hours. Is there any other procedure to transfer a file which is faster than FTP. Server Location: One in India and another in US.

share|improve this question

marked as duplicate by Scott Pack, Michael Hampton, mdpc, Ward, Dave M Feb 15 '13 at 13:51

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

3  
Your transfer speed is about 2 Mbit/s. How fast is the line between the two ends? –  grassroot Feb 14 '13 at 6:49
    
Nope. 2GB (giga-bytes base 2, more than 17 billion bits) transferred at 2Mb/s (mega-bits base 10) would take 8590 seconds, or 2 hours, 23 minutes. But.... when you include overhead (ethernet, tcp/ip and, shudders, FTP) it's more than 3 hours. I assumed (yes, I know what assuming does) a 20% overhead for FTP and the old stand-by 2 bit per byte for TCP/IP & Ethernet and get 3 hours 34 minutes. –  SuperMagic Feb 14 '13 at 20:15
add comment

8 Answers

up vote 1 down vote accepted

Is this normal for your location? If you transfer 2GB to other sites, what sort of speed do you usually see? What is the network speed at your location and at the remote location?

The transfer speed over long distances will mostly depend on the network bandwidth available between the two locations and for any hop in between. You are transferring data over the Internet which means that speeds can vary greatly from time to time. The Internet does not guarantee a minimum speed.

Your best bets are:

  1. Before transferring the data, compress the data on your local server so that they are smaller in size.
  2. Use rsync to compress the data on the fly. See the examples at http://en.wikipedia.org/wiki/Rsync#Examples
  3. Break the data into chunks which you transfer one at a time. This won't speed up the data transfer, but it will make the transfer more fault tolerant (you won't need to restart from the beginning if the transfer fails 99% of the way through). Compression can help.
  4. Another ISP or network may offer better transfer speeds. Try another ISP or network.
share|improve this answer
add comment
  • Log on to one of the servers using SSH
  • Open up file transfer from the server you're logged in on to the other server (e.g. lftp, scp, etc.)
  • Send the file.

The last two are problably the same thing, but I wanted to get 3 steps out of it. It's as simple as "just send the file."

share|improve this answer
    
I made a script which will transfer a file regularly. I don't want to use root password therefore I am not using SCP and ftp is taking so much time... any other solution ?? –  Himanshu Matta Feb 14 '13 at 6:40
2  
SCP works with any user account, not just root. –  tylerl Feb 14 '13 at 6:42
1  
SCP will not be any faster than FTP. On the contrary it includes encryption, which will slow down the transfer. –  grassroot Feb 14 '13 at 6:46
    
I think something is going wrong. Normally, we use FTP to transfer a file. But in this case it is taking so much time. So can you tell me what are the possible conditions that can affect FTP tranfer –  Himanshu Matta Feb 14 '13 at 6:52
add comment

When I was reading about file transfer performance for NAS devices, the speeds people were getting over SMB was typically always slower than FTP. I got the sense that whilst there might be a lot of things annoying about FTP, it's not in itself a slow protocol.

I wouldnt expect that, leaving everything else unchanged, merely changing from FTP to another protocol is going to give you the kind of speed improvement you seem to be expecting.

How do you know the problem isn't bandwidth? Can you try doing a test with scp or rsync and comparing with FTP?

share|improve this answer
    
A new twist in story .. I tested file transfer using FTP. If I am transferring small file from command line then it takes 1 or 2 second. But If I use command throuh php code then it takes 15 to 20 seconds. –  Himanshu Matta Feb 14 '13 at 7:50
add comment

Do you have TCP window scaling enabled at both ends? Even if you have a lot of bandwidth, connections between the U.S. and India will inherently have a huge bandwidth delay product. If window scaling is enabled and you're still seeing poor FTP performance you might try running rsync in parallel.

Also, stop using FTP. It's a horrible protocol. :)

share|improve this answer
add comment

SF community members may not be happy for taking you back to the stone age's of Unix, but for lower protocol overhead and a good compression, you might want to try a combination of dd + netcat + bzip2. Yes, this is not going to be secure, so you have to close the ports for all except for the two nodes. No guarantee, no security, no authentication ... but it is faster.

1 - compress your file using bzip2 to get say , file.bz2

2 - Listen using netcat on node2

     nc -l 6668 | dd of=/dir/file.bz2
  1. Push it from node1

     dd if=/dir/file.bz2 | nc node2 6668
    
share|improve this answer
add comment

If youre trying to transfer a single 2GB file ftp is one of the most clean and fast protocols, you can also safely use resume feature in case of interruption during the transfer.

Ftp is a bad idea if you're trying to transfer a great number of little files, because you have a huge useless activity opening and closing connections for each file.

Rsync(via ssh) or scp(sftp) are good alternatives, but you must consider that encryption significantly reduce throughput. You can use a light algorithm like blowfish ('scp -c blowfish' or 'rsync -e "ssh -c blowfish"') but the speed will always lower than ftp.

Netcat is an alternative, but only on lan.

share|improve this answer
add comment

If your file is similar to one already at the destination you could use rsync with the --fuzzy option. Fuzzy will look for a similarly named file in the destination directory and use it as the starting point for the copy sending only the differences between the two files across the wire.

As an example, here I'm copying xwiki-enterprise-web-3.4.war into a remote directory which already contains a xwiki-enterprise-web-3.2.1.war.

$ rsync -av --fuzzy xwiki-enterprise-web-3.4.war example.com:Downloads/xwiki-enterprise-web-3.4.war
sent 69065310 bytes  received 74176 bytes  389518.23 bytes/sec
total size is 138567168  speedup is 2.00

You'll notice only about 1/2 the contents of the file actually needed transfer. It would be even better if the files were more similar as in this specific case the newer version xwiki is actually 26M larger than the older copy.

$ du -sh xwiki-enterprise-web-3.*
107M    xwiki-enterprise-web-3.2.1.war
133M    xwiki-enterprise-web-3.4.war
share|improve this answer
add comment

Instead of that why not to use rsync or rdiff command both are fast tools to transfer files.

share|improve this answer
    
I dont have idea about these tools...are you sure if I'll use these tools then I'll be able to transfer files with good speed. –  Himanshu Matta Feb 14 '13 at 7:22
1  
Unless you're willing to invest in WAN optimization technology like Aspera or similar, you're stuck using gzip compression with rsync. –  Joel E Salas Feb 14 '13 at 7:33
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.