Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We wish to install some servers in a remote datacentre to act as a backup storage location for our primary datacentre.

Assuming that both sites will have GigE connectivity, what is the best method to use for fast file transfer? I love rsync, however since we have a lot of data to transfer (1.5TB per night), I think that the SSH protocol used in rsync may slow things down a lot :(

We could install some fast VPN endpoints to cater for link encryption, however the question still stands: what is the best tool for the actual transfer?

share|improve this question
    
your rate of change is 1.5TB per night? are you sure if you sent only deltas that would still by the case? –  SpacemanSpiff May 19 '12 at 13:01
    
Yes, we are a hosting company. –  jtnire May 19 '12 at 13:05
    
@jtnire: I don't think you understand what SpacemanSpiff was asking... –  womble May 19 '12 at 13:40
    
@womble Yes I did. What is there not to understand? When you are a hosting company that has many, many TB of customer data, it is perfectly reasonable to have 1.5TB of deltas per night –  jtnire May 19 '12 at 14:00
add comment

5 Answers

backup performance is determined by many factors. Bandwidth being one of them.

  • Storage write performance

Often determined by the storage write performance.

  • Network Bandwidth

A good option is to run rsync in daemon mode on the backup server, doing this you would avoid ssh. However unless you are having really slow processors, ssh overhead would not be significant.

To run rsync as daemon start rsync daemon on the server

  rsync --daemon 

By default it listens on TCP port 873 you can change it in rsyncd.conf.

Then use rsync as

 rsync [OPTIONS] source-path \
          rsync://backup_username@backup-server:873:destination-path

There is not enough information to give an estimation of your expected performance. Yet daily addition of 1.5 TB is not impossible.

  • Storage IOPS

During backup you combine write operations with a number of file system ops. Filesystem queries and updates. It is generally a good idea to run several rsync processes to hide the latency of file create.

share|improve this answer
    
Thanks for this. Can you please discribe how rsync daemon avoids SSH? Thanks –  jtnire May 19 '12 at 13:04
    
@jtnire rsync uses it's own non-encrypted protocol. I have added some details. –  Dmitri Chubarov May 19 '12 at 13:20
    
You could also look into making differential/incremental tars first (via timestamp files) and sending these offsite by rsync (or, if you really want to bother with orchestrating it, plain netcat pipes ;). This will save a lot of filesystem overhead (which can matter a lot indeed with such volumes). Paralleling rsyncs, btw, needs to be finely tuned - you don't want to make the receiving storage system start to thrash!, and you want to partition the data they each synchronize, a) because of conflicts possible and b) because of RAM usage) –  rackandboneman May 19 '12 at 22:55
add comment

You may want to look into file acceleration software. I think there are many players in this market, but the one I have seen used in the past was aspera. Here is a page comparing aspera sync to rsync (comparison tables at bottom of page).

http://asperasoft.com/en/products/synchronization_23/aspera_sync_23

share|improve this answer
    
BitSpeed appears to offer a similar product. –  David Schwartz May 19 '12 at 13:27
add comment

Also, make sure that no side involved uses any really old versions of rsync. There are still 2.x versions in use, these make the whole chain fall back to an older and in some cases far less efficient version of the protocol (If you are told "sending incremental file list", you are fine. If it is "sending file list", that is 2.x protocol used.)

share|improve this answer
add comment

I think 1,5 TB delta/day is a bit out of the typical size for solutions like rsync. SSH has a architectural cap at about 2-3MB/s IIRC and as written before the default rsync protocol is much faster but unencrypted.

You should really have a look at solutions which are specifically designed to synchronize these amounts of data. What I have worked with in the past are the Quantum DXi appliances which are storage appliances but also offer deduplication and encrypted replication. You might want to have a look at these.

/edit: To extend my above statement a bit more, it is important to take the following things into consideration when measuring SSH speed:

  • The Speed issue occurs because of the internal buffer structure of SSH because it was not originally developed to transfer large amounts of data over WAN (read here for more details and a possible solution
  • Take into account the RTT. Because of the buffer issue the performance over WAN (which is what the TO asks) can be much worse than on local gigabit, even when adding only 10ms RTT
  • Compressing: A hosting company will have a lot of files which cannot be compressed any more like downloads which are already compressed, movies, images etc. This slows overall throughput as you cannot count on the data to be reduced to 20% or less, I'd estimate you can calculate with 50% compression ratio tops.
  • Count of files/Compression: You obviously cannot create one single archive of 1,5TB and sync that. Why? Because if one byte of this archive is corrupt (due to whatever reason) the whole backup might be useless. So you'd have to split the deltas to maybe 1 archive per customer which adds overhead to the transfer and also worsens compression ratio

The big advantage on deduplication here would be that data is deduplicated on a block level. Meaning if you'd create one tar (not compressed!) per customer and put that an one of the DXi appliances at your main site this appliance automatically will eliminate duplicate blocks in the file stream (e.g. 100 customers have the same movie in their tar - it will only be stored once and will be referenced the other 99 times), and the blocks will also be compressed.

If you then add a second one off-site only the unique data blocks are transferred to the second appliance. With that you could in fact perform daily full backups at your main site and only the size of newly written unique blocks would have to be transferred over WAN to the off-site

share|improve this answer
    
what you write about ssh is incorrect. "rsync -av --progress root@server:/2GB.test ./" gives me without any tuning ~75MB/s transfer over a gigabit link. adding "-e 'ssh -c arcfour'" parameter gives me 106MB/s transfer. –  pQd May 20 '12 at 6:27
    
No it is not. I have added more details to my original post as it was to much for a comment. I guess that your test scenario did not factor in one or more of these points, e.g. the round trip time and maybe you also tested with non-random data which can be compressed very efficiently –  leepfrog May 20 '12 at 8:37
    
running things in parallel helps a lot to utilize available resources [network bandwidth, multicore cpus, io] helps a lot. i'm pushing tens and hundreds of gigabytes every night across wans. admittedly 100mbit/s not gigabit. but it's still quite easy to saturate that link. –  pQd May 20 '12 at 8:42
add comment

someone mentioned here using rsync daemon - this is much 'lighter' solution than tunneling the traffic over ssh. but even with ssh encapsulation transferring 1.5TB over night and saturating gigabit link should be doable.

assuming you have few large files [possibly wrong assumption] - you should be able to transfer the payload within ~5h. i've done a quick test:

server:/mnt/big/tmp# rsync -av --progress root@otherServer:/big/file ./
receiving incremental file list
file
  1849044309 100%   74.47MB/s    0:00:23 (xfer#1, to-check=0/1)

sent 30 bytes  received 1849270109 bytes  75480413.84 bytes/sec
total size is 1849044309  speedup is 1.00

telling ssh to use lighter compression method:

server:/mnt/big/tmp# rsync -e "ssh -c arcfour" -av --progress root@otherServer:/big/file ./
receiving incremental file list
file
  1849044309 100%  106.70MB/s    0:00:16 (xfer#1, to-check=0/1)

sent 30 bytes  received 1849270109 bytes  112076978.12 bytes/sec
total size is 1849044309  speedup is 1.00

so assuming storage is not a bottleneck - 106MB/s ~= 350GB/h ~= 1.5TB in 5h.

both tests were done on idle machine with xeon E5430 @ 2.66GHz cpu.

to get things more efficient [make use of multiple cores if you have slower CPU] or just use better available bandwidth and IO - you can run few parallel rsync sessions for a few files.

i dont know if you own/lease the fiber or just use mpls service provided by the operator regardless of those ssh gives you additional benefit of strong encryption without setting vpn in-between.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.