Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've got a site-to-site vpn connection between two data centers (one in San Jose, the other in Toronto).

I need to send a 32GB file from one dc to another - FAST AS POSSIBLE.

I've found a shell script that splices up the 32GB file smaller files and then uses scp to transfer over in parallel fashion.

The question is how do I determine the optimal file size for sending over the various little files across the site-to-site vpn connection (I'd like to try to maximize the bandwidth).

Obviously the more scp process I run on the server, I guess there's more load put on that server.

share|improve this question

3 Answers 3

up vote 1 down vote accepted

Forget about the site-to-site for a minute, because as long as its ipsec and your endpoints are not toasters its unlikely to be bottleneck in itself, and have a quick look at bbcp:

http://www.slac.stanford.edu/~abh/bbcp/

Here is a line from the perl script we used during the last migration, which had the same requirements as you i.e. move the data fast

sprintf('/usr/local/bin/bbcp -a -F -s 16 -P 10 -T "ssh -x -a -oFallBackToRsh=no %%I -l %%U %%H /usr/local/bin/bbcp" -d . -v %s %s:%s',
  join(' ', @files_to_copy), $remote_host, $destination_dir);

Play with the options, especially the number of threads.

Questions you will want answered are:

  • what is the latency of the links
  • what is the packet jitter likely to be like
  • what is the total maximum bandwidth I can expect
  • who/what else will I stomp on by hogging the entire link

bbcp should be able to max any link to the point where cpu becomes your bottleneck with the right flags. Good luck

share|improve this answer

I'd look at rsync for anything that large.

Something like:

rsync -ave "ssh -c arcfour -o Compression=no -x" source_file user@destination:/path/to/dest

http://en.wikipedia.org/wiki/RC4

That way, if a partial copy gets interrupted for any reason, you will be able to resume the upload leveraging rsync's internals.

share|improve this answer

If you're willing to consider commercial solutions, Aspera or Signiant are much faster than scp, sftp, or rsync.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.