How can I speed up file transfers between storage arrays?

Question

I have two large storage arrays from different vendors, each with about 10Gb/s of bandwidth available for read/write. I need to move multiple TB's of files from one to the other. Each speaks NFS and CIFS. Currently using cp or scp or rsync, the intermediate host's GbE interface becomes the bottleneck, since I have to read from one array and write to the other. There's no host on either array that I can log into to cut out the middleman. I have bunches of hosts that could provide additional bandwidth, if only I could do something like:

  rsync [src] [dest] --parallel-hosts=host1,host2,host3,host4

Is there a straightforward solution to this?

Answer 1

One trick I use for parallelizing things like this is doing it in two passes - first generate a list of files to transfer, then split that list, then farm the work out to multiple machines.

You can do this with rsync with something like this:

• first run rsync with --itemize-changes (this just trawls the filesystems and outputs a list of files which would be copied etc)
• process the --itemize-changes output to extract the filenames and split the list into chunks
• run each chunk in a separate rsync process (or on a separate machine) using rsync with --files-from=chunkXX

This helps massively for network filesystems (NFS, Lustre, etc) where you typically can't get enough performance from a single rsync process/machine.

It still takes a while to trawl the filesystems to compare them, but the actual data movement can be very quick. Also, things like cpio and cp tend to outperform rsync for raw data movement.

Another option I use pretty often is to do a simple find -type f /src and then split that list of files and feed it to cpio on multiple hosts.

I'd love to know of a simpler/quicker way to do this...

Answer 2

If the two storage arrays will not talk directly to each other, you're best option would be to build a purpose-built machine with high-bandwidth connections. Even a relatively inexpensive new workstation with a pair of 10GbE NICs can pump out a lot of bandwidth if the supporting network and storage arrays can take it.

Answer 3

You should talk to your storage vendors to determin if the controllers can talk directily to each other. If not build or rent a high bandwidth intermidate node. My guess is that there is a way to make the two arrays play nice and run the clone full bore on the controllers.

Answer 4

If you can't find a way for the controllers to talk to each other (as others have mentioned), you can try doing this:

On your destination server, run the following command:

$ nc -l -p 9999 | tar xvzf -

Then, on your source server, run the following command:

$ tar cvzf - <SOURCEFILES> | nc -q 1 <DST_SERVER> 9999

The advantage to this is it avoids any encryption overhead that SSH/rsync gives, so you'll get a bit of a speed boost. This also compresses and decompresses on the source and destination servers in-stream, so it speeds up the transfer process at the expense of some CPU cycles.

Good luck!

Answer 5

Mark,

What vendor is the source and which vendor is the target? And what models are they. I may be able to answer your question better from an internals point of view of the storage frame.