Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have some fairly large file transfers going across the atlantic every morning by ftp. From time to time the transfers will be distrupted, likely due connectivity issues, and the files will be left in an incomplete state.

I'm wondering if there is a better way to handle these file transfers. Would say, rsync or scp be more resilient to periodic loss of connectivity? The best would be if the transfer would reconnect and continue where it left off, should it be disrupted.

Large in this case means files up to several GiB and transfers spanning several hours.

share|improve this question
    
Doesn't FTP do those things? Perhaps you need to tweak the settings on the client/server? –  V. Romanov Nov 3 '09 at 8:28
    
How large is large? 500mb, 1gb, 1tb? –  Zoredache Nov 3 '09 at 8:29
    
The common ftp clients doesn't do this on their own, but it might be possible to script a framework. Eg. read the exit code, if not 0 restart the transfer. –  Roy Nov 3 '09 at 8:40
    
The daily amount of data is in 10-50 GB range. –  Roy Nov 3 '09 at 8:41

6 Answers 6

up vote 10 down vote accepted

I belevei this is a case where rsync should work well. Rsync does a very good job at resuming where it left off when a transfer is incomplete. If an older version of the file already exists on the remote side rsync is really good at only transmitting the differences.

I don't really think scp is what you want. If you need to transmit over ssh, then you can pass the option -e ssh to rsync and ssh will be used as the transport.

share|improve this answer
    
+1 for rsync. It happened to me yesterday when my wifi connection got cut off for a while. The transfer resumed after that. –  sybreon Nov 3 '09 at 8:56

I second the rsync option, however if you want it to resume a transfer of a big file, then don't forget to use the --partial command-line option which will mean it doesn't delete the temporary file it was writing to when the link goes down. You'll probably have to call it in a loop which retries when the link goes down.

Something like

until rsync --partial -e ssh sourcefile remote:destination; do 
 echo Retrying; 
done
share|improve this answer
    
Thats a very elegant loop! –  Roy Nov 3 '09 at 11:09

Why not BitTorrent? You get multiple bonuses if you use it. At least reliable transfers and availability to seed from multiple locations.

share|improve this answer
1  
It's a point to point transfer of private data. Bittorrent is good for many things, but probably not for this. –  Roy Nov 3 '09 at 11:08
    
You can encrypt it before transferring. –  Anonymous Nov 6 '09 at 16:52

Definitely use rsync with -P - but for better performance over the Atlantic you probably want to use one of the UDP based file transfer tools - Aspera, FileCatalyst - or the open source Tsunami UDP.

share|improve this answer
    
Thats an interesting idea. Even though I'd prefer to use standard utilities, I'll have a good look at these. –  Roy Nov 3 '09 at 14:01

Maybe Direct Connect (or other P2P protocol) is the best choice for you?

share|improve this answer
    
I don't think a p2p file sharing solution is appropriate for these transfers. Thanks anyway. –  Roy Nov 3 '09 at 14:02

Have you checked out accellion? They are a ftp/webmail hybred. We use it for our graphics designers to securely and easily send large files to the printers.

They offer a free virtual appliance. It also has encryption, delivery notification and forwarding prevention, 2GB max file size, all built-in for free.

If you pay you can setup automation and allow up to 10GB files.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.