At the moment i'm running rsync for 2.2 million files total of 250GB and that just takes ages 700K files in 6 hours.
Does anyone know a rsync like tool that can do this with multiple threads so it goes faster?
At the moment i'm running rsync for 2.2 million files total of 250GB and that just takes ages 700K files in 6 hours.
Does anyone know a rsync like tool that can do this with multiple threads so it goes faster?
I doubt cpu is the limiting factor here. You're most likely limited by both network bandwidth for the transfer, and disk IO; especially latency for all those stat calls.
Can you break down the filesystem hierarchy into smaller chunks to process in parallel?
What are the source files, and what's writing or modifying them? Would it be possible to send changes as they happen at the application level?
If the disk subsystem of the receiving server is an array with multiple disks, running multiple rsync processes can improve performance. I am running 3 rsync processes to copy files to an NFS server (RAID6 with 6 disks per raid group) to saturate Gigabit Ethernet.
This guy reports on a basic python harness that spawns multiple rsync processes http://www.reliam.com/company/featured_geek
I've read many questions similar to this. I think the only real answer is break up the copy/move manually. IOps will be the issue here. If it makes you feel any better, I'm in the process of moving ~200 milllion files consuming well over 100TB of disk space.
You may consider checking out the multithreaded cp clone for linux (open source): http://static.usenix.org/event/lisa10/tech/slides/kolano.pdf