Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

At the moment i'm running rsync for 2.2 million files total of 250GB and that just takes ages 700K files in 6 hours.

Does anyone know a rsync like tool that can do this with multiple threads so it goes faster?

share|improve this question
3  
Have you any reason to assume it's CPU-bound in any way? –  Chopper3 Jun 10 '11 at 14:09
    
What rsync options are you using? –  Kyle Smith Jun 10 '11 at 14:14
    
Are you using ssh as a transport? –  JimB Jun 10 '11 at 14:16
    
rsync -avSPp And there are no CPU nor Disk issues. –  Tom van Ommen Jun 10 '11 at 14:16
    
And no SSH transport just saw something on the web dont know if its faster. It already takes ages to index all files. –  Tom van Ommen Jun 10 '11 at 14:17

4 Answers 4

up vote 7 down vote accepted

I doubt cpu is the limiting factor here. You're most likely limited by both network bandwidth for the transfer, and disk IO; especially latency for all those stat calls.

Can you break down the filesystem hierarchy into smaller chunks to process in parallel?

What are the source files, and what's writing or modifying them? Would it be possible to send changes as they happen at the application level?

share|improve this answer
    
Syncing Zarafa Attachment files, all gzipped by default. i could run multiple instances but thats less efficient than 10 threads. And the network is 1GBit to 1GBit but different datacenters but it shouldnt be a issue. got 24 SAS disks on the source side and intelligent storage with SSD on the destination. –  Tom van Ommen Jun 10 '11 at 14:20
1  
@Tom van Ommen - why do you think you're CPU limited? How is multiple processes less efficient than threads if you really are CPU limited? –  JimB Jun 10 '11 at 14:31
1  
@Tom van Ommen, 10 processes do have more overhead than 10 threads; however, locking data structures between threads is a coding nightmare. It's often much more efficient (for the coder's time) to just spawn multiple processes and be done with it –  Mike Pennington Jun 10 '11 at 14:36
1  
@Guacamole - multiple thread could help in some situations, but if his link is saturated, he's not going to push any more through no matter how many thread he has. Rsync does use threads for concurrency, and isn't internally blocking on IO. –  JimB Jun 10 '11 at 14:40
1  
@Guacamole - All I'm pointing out is that if he's using ssh as a transport, his throughput is limited by ssh itself (specifically the static receive window, unless he's using the HPN ssh patches). –  JimB Jun 10 '11 at 15:28

If the disk subsystem of the receiving server is an array with multiple disks, running multiple rsync processes can improve performance. I am running 3 rsync processes to copy files to an NFS server (RAID6 with 6 disks per raid group) to saturate Gigabit Ethernet.

This guy reports on a basic python harness that spawns multiple rsync processes http://www.reliam.com/company/featured_geek

share|improve this answer

You may consider checking out the multithreaded cp clone for linux (open source): http://static.usenix.org/event/lisa10/tech/slides/kolano.pdf

share|improve this answer
    
Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. –  Scott Pack Oct 19 '12 at 22:48

I've read many questions similar to this. I think the only real answer is break up the copy/move manually. IOps will be the issue here. If it makes you feel any better, I'm in the process of moving ~200 milllion files consuming well over 100TB of disk space.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.