I have two large storage arrays from different vendors, each with about 10Gb/s of bandwidth available for read/write. I need to move multiple TB's of files from one to the other. Each speaks NFS and CIFS. Currently using cp or scp or rsync, the intermediate host's GbE interface becomes the bottleneck, since I have to read from one array and write to the other. There's no host on either array that I can log into to cut out the middleman. I have bunches of hosts that could provide additional bandwidth, if only I could do something like:

  rsync [src] [dest] --parallel-hosts=host1,host2,host3,host4

Is there a straightforward solution to this?

  • Edited: 10T/s -> 10Tb/s – Mark Maxham Aug 18 '10 at 19:11
  • 10 terabits per second? Even 10 gigabits would be pretty ferocious as storage goes. – Skyhawk Aug 18 '10 at 20:56
  • Sorry, brain fart. I meant 10Gb/s. Fixed. – Mark Maxham Aug 18 '10 at 23:01
  • Whew! That makes more sense. 10Tbps would take about 18,000 WDC Raptor Hard Drives (Rated at 72MB/s) and 34 12X ERD Infiniband connections on each array! – Chris S Aug 18 '10 at 23:02
  • How do you talk to the storage arrays? NFS? – James Aug 18 '10 at 23:07

One trick I use for parallelizing things like this is doing it in two passes - first generate a list of files to transfer, then split that list, then farm the work out to multiple machines.

You can do this with rsync with something like this:

  • first run rsync with --itemize-changes (this just trawls the filesystems and outputs a list of files which would be copied etc)
  • process the --itemize-changes output to extract the filenames and split the list into chunks
  • run each chunk in a separate rsync process (or on a separate machine) using rsync with --files-from=chunkXX

This helps massively for network filesystems (NFS, Lustre, etc) where you typically can't get enough performance from a single rsync process/machine.

It still takes a while to trawl the filesystems to compare them, but the actual data movement can be very quick. Also, things like cpio and cp tend to outperform rsync for raw data movement.

Another option I use pretty often is to do a simple find -type f /src and then split that list of files and feed it to cpio on multiple hosts.

I'd love to know of a simpler/quicker way to do this...

  • This was what I was looking for ... except I was hoping for a one-line solution. I can code that up easily enough though. – Mark Maxham Aug 19 '10 at 4:31

If the two storage arrays will not talk directly to each other, you're best option would be to build a purpose-built machine with high-bandwidth connections. Even a relatively inexpensive new workstation with a pair of 10GbE NICs can pump out a lot of bandwidth if the supporting network and storage arrays can take it.

  • I'll see if we have any machines with 10GbE interfaces. – Mark Maxham Aug 18 '10 at 23:03
  • Nope, no 10G in the house. I guess I could upgrade a host. – Mark Maxham Aug 19 '10 at 0:52
  • Even with 10 gbit NICs, there is no way you will see anything close to wire speed throughput via NFS/CIFS unless you have multiple parallel connections. – James Aug 19 '10 at 19:52

You should talk to your storage vendors to determin if the controllers can talk directily to each other. If not build or rent a high bandwidth intermidate node. My guess is that there is a way to make the two arrays play nice and run the clone full bore on the controllers.


If you can't find a way for the controllers to talk to each other (as others have mentioned), you can try doing this:

On your destination server, run the following command:

$ nc -l -p 9999 | tar xvzf -

Then, on your source server, run the following command:

$ tar cvzf - <SOURCEFILES> | nc -q 1 <DST_SERVER> 9999

The advantage to this is it avoids any encryption overhead that SSH/rsync gives, so you'll get a bit of a speed boost. This also compresses and decompresses on the source and destination servers in-stream, so it speeds up the transfer process at the expense of some CPU cycles.

Good luck!

  • I don't know of many compression algorithms that can pump out 10Gbps with even a modern processor. Compression is likely to slow it down. – Chris S Aug 19 '10 at 2:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.