Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Possible Duplicate:
what is the fastest and most reliable way of transferring a lot of files?

I am currently attempting to transfer over 1 million files from one server to another. Using wget, it seems to be extremely slow, probably because it starts a new transfer after the previous one has been completed.

Question: Is there a faster non-blocking (asynchronous) way to do the transfer? I do not have enough space on the first server to compress the files into tar.gz and transferring them over. Thanks!

share|improve this question

marked as duplicate by Michael Hampton, MadHatter, Ladadadada, mulaz, rnxrx Oct 15 '12 at 17:55

This question was marked as an exact duplicate of an existing question.

Do you mean "in parallel" rather than "asynchronous"? Also, the physical distance between the servers, the bandwidth between the servers and the total size of the files will all affect any answers. The latency between the servers will only affect some solutions and can be worked around. – Ladadadada Oct 10 '12 at 14:59
Yes in parallel will work too. I believe theres at least a 100Mbps connection between them, ping is 80ms, each file size is under 50KB, between 1-1.5 million of them. All the files are placed in individual folders by date. I could eventually start deleting the folders whose contents have been transferred, compress the other folders one at a time and send it over. – Nyxynyx Oct 10 '12 at 15:01
Use rsync. It allows to copy a whole bunch of files. – Eun Oct 10 '12 at 15:10
up vote 0 down vote accepted
  • Run 1 rsync process per directory, until you saturate your network link. Script it so that a new rsync process is triggered when the previous one ends.
  • or, run 1 rsync process per unique character at the start of the filename using includes.
  • or, run 1 rsync process per unique 1st + 2nd character combination of the filename using includes.

Basically rsync - doing whatever it takes to trigger enough to saturate your network link.

share|improve this answer

Put the files on a hard drive and ship it via FedEx, UPS, DHL, etc.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.