up vote 2 down vote favorite
share [g+] share [fb]

I have about 6 TB of files I need to move from one server to another. I did my best to move it over FTP like below, but the connection dies a lot, and after a certain amount of progress, it disconnects before it even resumes moving files from I presume taking too long to compare files before actually transferring and then timing out. 

~/ncftp-3.2.3/bin/ncftpput -R -z -v -u "user" -p "password" upload.server.net /local/dir/ remote/dir/

I'm trying to get the remote server to give me access with SSH so I can set up an rsync, but is there anything more stable I can do over ftp, so maybe it'll try on it's own to resume without recomparing the entire file lists?

link|improve this question
Can you supply some more details about the network setup between the servers? Network speed? Latency? Internal network or publicly over Internet? – Anders Abel Mar 7 '10 at 20:11
feedback

3 Answers

up vote 4 down vote accepted

If your only access is via FTP, you may want to look into lftp (should be in most distros).

lftp supports automatic retries on failure and also has a mirror option which sounds like it matches what you want to do.

link|improve this answer
+1 for lftp. I've used it for years and it's great. And nobody I mention it knows it :(. – chmeee Mar 7 '10 at 21:20
Fastest method I found was using lftp after Tarring up the files into a couple hundred mb packages. – UltimateBrent Jun 18 '10 at 6:43
feedback
up vote 1 down vote

i'd use rsync. if connection drops it'd compare source and destination and sync from where it left (assuming large amount of small to medium files, not 2 x 3 TB :) ).

alternatively start apache and make your file dir root and do recursive wget, might work as well, you just need to tell it to ignore files that already exist locally.

link|improve this answer
rsync would resume 2 X 3TB just fine, though it would be slow unless you used the --append option. – Justin Mar 7 '10 at 22:38
feedback
up vote 0 down vote

You don't say what the boundaries are for this project. I myself in my situation would do this by backing up the data from the source and restore it to the destination using my backup software.

If you can't do things like that then why not try moving the data in "batches" if that's possible, if the data is split up into a number of files or directories?

link|improve this answer
What boundaries are you looking for? The point is I want it to be able to resume itself without me having to setup 300 separate batches. – UltimateBrent Mar 7 '10 at 19:13
5  
"Never underestimate the bandwidth of a station wagon filled with mag tape." Or maybe a box of Blu-Ray RW discs. Seriously, for 6 TB of data, backup/restore (with perhaps a FedEx step inbetween) is not unreasonable for hosts not on the same LAN. If they are on the same LAN, then one should already have backups available locally. – mpez0 Mar 7 '10 at 19:29
That's what I'm thinking at this point, looks like i'll fall back on lftp if not. – UltimateBrent Mar 7 '10 at 21:25
@UltimateBrent - 300 batch jobs - I hear that, but if it absolutely positively must be completed by thursday morning, say, then sometimes you have to be pragmatic about these things right? – DJ Pon3 Mar 8 '10 at 6:34
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.