Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Possible Duplicate:
what is the fastest and most reliable way of transferring a lot of files?

I am currently attempting to transfer over 1 million files from one server to another. Using wget, it seems to be extremely slow, probably because it starts a new transfer after the previous one has been completed.

Question: Is there a faster non-blocking (asynchronous) way to do the transfer? I do not have enough space on the first server to compress the files into tar.gz and transferring them over. Thanks!

share|improve this question
    
Do you mean "in parallel" rather than "asynchronous"? Also, the physical distance between the servers, the bandwidth between the servers and the total size of the files will all affect any answers. The latency between the servers will only affect some solutions and can be worked around. –  Ladadadada Oct 10 '12 at 14:59
    
Yes in parallel will work too. I believe theres at least a 100Mbps connection between them, ping is 80ms, each file size is under 50KB, between 1-1.5 million of them. All the files are placed in individual folders by date. I could eventually start deleting the folders whose contents have been transferred, compress the other folders one at a time and send it over. –  Nyxynyx Oct 10 '12 at 15:01
    
Use rsync. It allows to copy a whole bunch of files. –  Eun Oct 10 '12 at 15:10
add comment

marked as duplicate by Michael Hampton, MadHatter, Ladadadada, mulaz, rnxrx Oct 15 '12 at 17:55

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

2 Answers

up vote 0 down vote accepted
  • Run 1 rsync process per directory, until you saturate your network link. Script it so that a new rsync process is triggered when the previous one ends.
  • or, run 1 rsync process per unique character at the start of the filename using includes.
  • or, run 1 rsync process per unique 1st + 2nd character combination of the filename using includes.

Basically rsync - doing whatever it takes to trigger enough to saturate your network link.

share|improve this answer
add comment

Put the files on a hard drive and ship it via FedEx, UPS, DHL, etc.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.