Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 7 down vote favorite
2

I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100GB != 10TB).

I've got about 10TB that I need to move from one raid to another, gigabit net, XFS file systems. My biggest concern is having the transfer die midway and not being able to resume easily. Speed would be nice, but ensuring transfer is much more important.

Normally I'd just tar & netcat, but the raid I'm moving from has been super flaky as of late and I need to be able to recover and resume if it drops mid process. Should I be looking at rsync?

EDIT: looking into this a bit more, i think rsync might be too slow, i'd like to avoid this taking 30+ days. So now i'm looking for suggestions on how to monitor / resume the transfer with netcat.

share|improve this question
I sometimes need to do large file transfers (But not 10TB ;) rsync has many flags, some of these might impact the performance of a large file transfer (I think --checksum and --archive might slow you down, for example. This would make a big difference when transferring 10TB. ). Can anyone recommend good options to help optimize the performance of such a large file transfer? Would tuning --block-size=SIZE help? – Stefan Lasiewski Jun 7 '10 at 22:24
is there anyway to remove the ssh overhead? – lostincode Jun 7 '10 at 22:45
1  
set up rsyncd on your receiving end? no need for ssh – cpbills Jun 7 '10 at 23:12
3  
Run an rsync daemon on the receiving side as well as the client on the sending side. – Dennis Williamson Jun 7 '10 at 23:13
2  
If you can't set up an rsync daemon on one side and are stuck with SSH, you can reduce the encryption overhead with less-good encryption like: rsync -avz -e 'ssh -c arcfour' SOURCE DEST – Dave Jun 8 '10 at 3:14
show 2 more comments

3 Answers

up vote 10 down vote accepted

yep, rsync

outside oddball, the async features DRBD came out with recently.

share|improve this answer
1  
+1 for Rsync. King of system-to-system transfers on linux. – GruffTech Jun 7 '10 at 22:37
1  
+1 rsync ftw... – Marco Ramos Jun 8 '10 at 1:12
+1, rsync (and it's Windows counterpart DeltaCopy) are the end all of efficient server to server generic file duplication. – Chris S♦ Jun 8 '10 at 2:29
1  
+1 for rsync, but I'd add that it may be faster to do it "locally" over an NFS mount instead of incurring the rsync or SSH protocol overhead. – James Sneeringer Jun 8 '10 at 17:45
up vote 0 down vote

You could try setting up an FTP server on the server with the data to be copied and use an FTP client with "resume" on the receiving end. I use Filezilla server and client and I use the "resume" feature of the client quite often and it has always worked without a hitch.

share|improve this answer
up vote 0 down vote

Never underestimate the bandwidth of a station wagon full of tapes. 10TB would be feasible with relatively cheap consumer grade NAS equipment if you can divide it into (say 2TB) chunks. If this is a one-off then a semi-manual process might be workable, and a 2TB NAS is only a few hundred dollars.

If you need an ongoing process then you could set up RSYNC after you've done the initial transfer.

share|improve this answer
The original question mentioned that this is already over a local network. – Ladadadada Jun 8 '10 at 16:44
Which appears to be too slow and/or unreliable for the OP's purposes. – ConcernedOfTunbridgeWells Jun 9 '10 at 8:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.