28

I have two Dell R515 servers running CentOS 6.5, with one of the broadcom NICs in each directly attached to the other. I use the direct link to push backups from the main server in the pair to the secondary every night using rsync over ssh. Monitoring the traffic, I see throughput of ~2MBps, which is much less than I'd expect from a gigabit port. I've set the MTU to 9000 on both sides, but that didn't seem to change anything.

Is there a recommended set of settings and optimizations that would take me to the maximum available throughput? Moreover, since I am using rsync over ssh (or potentially just NFS) to copy millions of files (~6Tb of small files - a huge Zimbra mailstore), the optimizations I am looking for might need to be more specific for my particular use case.

I am using ext4 on both sides, if that matters

Thanks

EDIT: I've used the following rsync options with pretty much similar results:

rsync -rtvu --delete source_folder/ destination_folder/

rsync -avHK --delete --backup --backup-dir=$BACKUPDIR source_folder/ destination_folder/

Currently, I'm looking at the same level of bad performance when using cp to an NFS export, over the same direct cable link.

EDIT2: after finishing the sync, I could run iperf and found performance was around 990Mbits/sec, the slowness was due to the actual dataset in use.

Improve this question
6
  • 1
    You should add rsync to your tags. Did you check the time for the listing part of rsync ? The low throughput can be due to small files. Can you post your rsync command to check options ?
    – kranteg
    Apr 20, 2014 at 18:15
  • @kranteg please see edit
    – dyasny
    Apr 20, 2014 at 18:57
  • 2
    Please verify connectivity with iperf.
    – ewwhite
    Apr 20, 2014 at 20:09
  • yup, iperf shows 991mbits/s, I guess it's te dataset that was so slow
    – dyasny
    Apr 21, 2014 at 21:08
  • You cannot have good throuphput with rsync and a dataset with small files. You should definitely try tar.
    – kranteg
    Apr 22, 2014 at 11:47

3 Answers 3

Reset to default
27

The file count and SSH encryption overhead are likely the biggest barriers. You're not going to see wire-speed on a transfer like this.

Options to improve include:

  • Using rsync+SSH with a less costly encryption algorithm (e.g. -e "ssh -c arcfour")
  • Eliminating encryption entirely over the SSH transport with something like HPN-SSH.
  • Block-based transfers. Snapshots, dd, ZFS snapshot send/receive, etc.
  • If this is a one-time or infrequent transfer, using tar, netcat (nc), mbuffer or some combination.
  • Check your CentOS tuned-adm settings.
  • Removing the atime from your filesystem mounts. Examining other filesystem mount options.
  • NIC send/receive buffers.
  • Tuning your rsync command. Would -W, the whole-files option make sense here? Is compression enabled?
  • Optimize your storage subsystem for the type of transfers (SSDs, spindle-count, RAID controller cache.)
Improve this answer
11
  • I've dumped SSH for NFS, seeing pretty much the same results. Block based transfers is what I am planning, switch to LVM snapshot based backups and dd the backups to the second server, where I'll be running ZFS for dedupe. atime is disabled on both sides. No compression is used. How do I optimize the storage subsys for this kind of transfer? The source has two RAID10 over 12x 10k SAS drives, one on the local drives, the other an MD1220. The backup server has the same disk count, but with large SATA drives, and uses RAID5. Full cache H800 and H700 controllers on both sides. 2MBps (from iftop)~
    – dyasny
    Apr 20, 2014 at 19:01
  • ~makes me think networking is the bottleneck here nonetheless.
    – dyasny
    Apr 20, 2014 at 19:01
  • @dyasny Test your network with iperf to be sure.
    – ewwhite
    Apr 20, 2014 at 19:01
  • 1
    @dyasny Yes. Most of what you need to know about ZFS is here
    – ewwhite
    Apr 20, 2014 at 20:53
  • 1
    Make sure the destination directory structure was created by rsync and not by cp. I've seen rsync take much longer to update a remote directory tree originally created by cp: 88GB updated with checksumming in 1h26m instead of 3h! How you create the initial disk layout is critical to getting good update performance. The CPU time is the same; the real time can double. (The same update without checkumming runs in 13 minutes from an SSD to a 200GB Seagate).
    – Ian D. Allen
    Apr 21, 2014 at 4:04
4

As you probably know copying a lot of little files (eg mailboxes using MailDir format or similar) is definitely not the best option to take advantage of high bandwith interfaces. SSH is probably not the best transport protocol for that either. I would try using tar to create a tarball on the source host prior to send it to you secondary host. 

tar c /var/mail | ssh root@secondary-host 'tar x -C /var/backups'

If you need incremental backup you may want to try the -g options of tar. If you still need to maximize throuput, try using netcat instead of ssh.

Improve this answer
5
  • I've switched to NFS instead of SSH, to remove the encryption overhead, no joy
    – dyasny
    Apr 20, 2014 at 19:02
  • Have you tried using tar? May be as a first step try creating a local tarbal on the primary server and then transfer it over the wire. (or test your network with iperf like @ewwhite suggeted)
    – alxgomz
    Apr 20, 2014 at 19:13
  • I would, if I had local space to spare. This is pretty huge, even with a fully populated DAS box
    – dyasny
    Apr 20, 2014 at 19:17
  • then try piping it over netcat or ssh (not this is as efficient though)
    – alxgomz
    Apr 20, 2014 at 19:18
  • I'll be switching to block based backups later on, and I intend to pipe dd through nc then. but right now, I'm stuck with two huge backups then need to be moved off of the main host, so I can create an LVM system there
    – dyasny
    Apr 20, 2014 at 19:20
1

Try teasing apart the contributing factors:

  • CPU (e.g. dd of /dev/zero piped through loopback)
  • disk I/O (e.g. dd of a large file piped to cat > /dev/null [piped to prevent short-circuiting])
  • physical network I/O (e.g. dd piped to the other machine)
  • etc.

and testing them independently.

I've had some bad experiences with Broadcom drivers, so my first suggestion is to test the usable network bandwidth with: dd if=/dev/zero bs=1m count=10k | rsh backup_host cat \> /dev/null

Improve this answer
1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .