Possible Duplicate:
what is the fastest and most reliable way of transferring a lot of files?

I am currently attempting to transfer over 1 million files from one server to another. Using wget, it seems to be extremely slow, probably because it starts a new transfer after the previous one has been completed.

Question: Is there a faster non-blocking (asynchronous) way to do the transfer? I do not have enough space on the first server to compress the files into tar.gz and transferring them over. Thanks!

  • Do you mean "in parallel" rather than "asynchronous"? Also, the physical distance between the servers, the bandwidth between the servers and the total size of the files will all affect any answers. The latency between the servers will only affect some solutions and can be worked around.
    – Ladadadada
    Oct 10, 2012 at 14:59
  • Yes in parallel will work too. I believe theres at least a 100Mbps connection between them, ping is 80ms, each file size is under 50KB, between 1-1.5 million of them. All the files are placed in individual folders by date. I could eventually start deleting the folders whose contents have been transferred, compress the other folders one at a time and send it over.
    – Nyxynyx
    Oct 10, 2012 at 15:01
  • Use rsync. It allows to copy a whole bunch of files.
    – Eun
    Oct 10, 2012 at 15:10

2 Answers 2

  • Run 1 rsync process per directory, until you saturate your network link. Script it so that a new rsync process is triggered when the previous one ends.
  • or, run 1 rsync process per unique character at the start of the filename using includes.
  • or, run 1 rsync process per unique 1st + 2nd character combination of the filename using includes.

Basically rsync - doing whatever it takes to trigger enough to saturate your network link.


Put the files on a hard drive and ship it via FedEx, UPS, DHL, etc.

Not the answer you're looking for? Browse other questions tagged or ask your own question.