Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two Dell R515 servers running CentOS 6.5, with one of the broadcom NICs in each directly attached to the other. I use the direct link to push backups from the main server in the pair to the secondary every night using rsync over ssh. Monitoring the traffic, I see throughput of ~2MBps, which is much less than I'd expect from a gigabit port. I've set the MTU to 9000 on both sides, but that didn't seem to change anything.

Is there a recommended set of settings and optimizations that would take me to the maximum available throughput? Moreover, since I am using rsync over ssh (or potentially just NFS) to copy millions of files (~6Tb of small files - a huge Zimbra mailstore), the optimizations I am looking for might need to be more specific for my particular use case.

I am using ext4 on both sides, if that matters

Thanks

EDIT: I've used the following rsync options with pretty much similar results:

rsync -rtvu --delete source_folder/ destination_folder/

rsync -avHK --delete --backup --backup-dir=$BACKUPDIR source_folder/ destination_folder/

Currently, I'm looking at the same level of bad performance when using cp to an NFS export, over the same direct cable link.

EDIT2: after finishing the sync, I could run iperf and found performance was around 990Mbits/sec, the slowness was due to the actual dataset in use.

share|improve this question
1  
You should add rsync to your tags. Did you check the time for the listing part of rsync ? The low throughput can be due to small files. Can you post your rsync command to check options ? –  kranteg Apr 20 at 18:15
    
@kranteg please see edit –  dyasny Apr 20 at 18:57
1  
Please verify connectivity with iperf. –  ewwhite Apr 20 at 20:09
    
yup, iperf shows 991mbits/s, I guess it's te dataset that was so slow –  dyasny Apr 21 at 21:08
    
You cannot have good throuphput with rsync and a dataset with small files. You should definitely try tar. –  kranteg Apr 22 at 11:47
show 1 more comment

3 Answers 3

up vote 4 down vote accepted

The file count and SSH encryption overhead are likely the biggest barriers. You're not going to see wire-speed on a transfer like this.

Options to improve include:

  • Using rsync+SSH with a less costly encryption algorithm (e.g. -e "ssh -c arcfour")
  • Eliminating encryption entirely over the SSH transport with something like HPN-SSH.
  • Block-based transfers. Snapshots, dd, ZFS snapshot send/receive, etc.
  • If this is a one-time or infrequent transfer, using tar, netcat (nc), mbuffer or some combination.
  • Check your CentOS tuned-adm settings.
  • Dumping the atime from your filesystem mounts. Examining other filesystem mount options.
  • NIC send/receive buffers.
  • Tuning your rsync command. Would -W, the whole-files option make sense here? Is compression enabled?
  • Optimize your storage subsystem for the type of transfers (SSDs, spindle-count, RAID controller cache.)
share|improve this answer
    
I've dumped SSH for NFS, seeing pretty much the same results. Block based transfers is what I am planning, switch to LVM snapshot based backups and dd the backups to the second server, where I'll be running ZFS for dedupe. atime is disabled on both sides. No compression is used. How do I optimize the storage subsys for this kind of transfer? The source has two RAID10 over 12x 10k SAS drives, one on the local drives, the other an MD1220. The backup server has the same disk count, but with large SATA drives, and uses RAID5. Full cache H800 and H700 controllers on both sides. 2MBps (from iftop)~ –  dyasny Apr 20 at 19:01
    
~makes me think networking is the bottleneck here nonetheless. –  dyasny Apr 20 at 19:01
    
@dyasny Test your network with iperf to be sure. –  ewwhite Apr 20 at 19:01
    
(and you probably don't want to use ZFS dedupe here. A world of pain awaits, especially for a dataset that size) –  ewwhite Apr 20 at 19:02
1  
@dyasny Yes. Most of what you need to know about ZFS is here –  ewwhite Apr 20 at 20:53
show 4 more comments

As you probably know copying a lot of little files (eg mailboxes using MailDir format or similar) is definitely not the best option to take advantage of high bandwith interfaces. SSH is probably not the best transport protocol for that either. I would try using tar to create a tarball on the source host prior to send it to you secondary host.

tar c /var/mail | ssh root@secondary-host 'tar x -C /var/backups'

If you need incremental backup you may want to try the -g options of tar. If you still need to maximize throuput, try using netcat instead of ssh.

share|improve this answer
    
I've switched to NFS instead of SSH, to remove the encryption overhead, no joy –  dyasny Apr 20 at 19:02
    
Have you tried using tar? May be as a first step try creating a local tarbal on the primary server and then transfer it over the wire. (or test your network with iperf like @ewwhite suggeted) –  alxgomz Apr 20 at 19:13
    
I would, if I had local space to spare. This is pretty huge, even with a fully populated DAS box –  dyasny Apr 20 at 19:17
    
then try piping it over netcat or ssh (not this is as efficient though) –  alxgomz Apr 20 at 19:18
    
I'll be switching to block based backups later on, and I intend to pipe dd through nc then. but right now, I'm stuck with two huge backups then need to be moved off of the main host, so I can create an LVM system there –  dyasny Apr 20 at 19:20
add comment

Try teasing apart the contributing factors:

  • CPU (e.g. dd of /dev/zero piped through loopback)
  • disk I/O (e.g. dd of a large file piped to cat > /dev/null [piped to prevent short-circuiting])
  • physical network I/O (e.g. dd piped to the other machine)
  • etc.

and testing them independently.

I've had some bad experiences with Broadcom drivers, so my first suggestion is to test the usable network bandwidth with: dd if=/dev/zero bs=1m count=10k | rsh backup_host cat \> /dev/null

share|improve this answer
    
Or iperf... –  ewwhite Apr 21 at 3:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.