9

At the moment i'm running rsync for 2.2 million files total of 250GB and that just takes ages 700K files in 6 hours.

Does anyone know a rsync like tool that can do this with multiple threads so it goes faster?

12
  • 3
    Have you any reason to assume it's CPU-bound in any way?
    – Chopper3
    Jun 10, 2011 at 14:09
  • What rsync options are you using?
    – Kyle Smith
    Jun 10, 2011 at 14:14
  • Are you using ssh as a transport?
    – JimB
    Jun 10, 2011 at 14:16
  • rsync -avSPp And there are no CPU nor Disk issues. Jun 10, 2011 at 14:16
  • And no SSH transport just saw something on the web dont know if its faster. It already takes ages to index all files. Jun 10, 2011 at 14:17

4 Answers 4

7

I doubt cpu is the limiting factor here. You're most likely limited by both network bandwidth for the transfer, and disk IO; especially latency for all those stat calls.

Can you break down the filesystem hierarchy into smaller chunks to process in parallel?

What are the source files, and what's writing or modifying them? Would it be possible to send changes as they happen at the application level?

12
  • Syncing Zarafa Attachment files, all gzipped by default. i could run multiple instances but thats less efficient than 10 threads. And the network is 1GBit to 1GBit but different datacenters but it shouldnt be a issue. got 24 SAS disks on the source side and intelligent storage with SSD on the destination. Jun 10, 2011 at 14:20
  • 1
    @Tom van Ommen - why do you think you're CPU limited? How is multiple processes less efficient than threads if you really are CPU limited?
    – JimB
    Jun 10, 2011 at 14:31
  • 1
    @Tom van Ommen, 10 processes do have more overhead than 10 threads; however, locking data structures between threads is a coding nightmare. It's often much more efficient (for the coder's time) to just spawn multiple processes and be done with it Jun 10, 2011 at 14:36
  • 1
    @Guacamole - multiple thread could help in some situations, but if his link is saturated, he's not going to push any more through no matter how many thread he has. Rsync does use threads for concurrency, and isn't internally blocking on IO.
    – JimB
    Jun 10, 2011 at 14:40
  • 1
    @Guacamole - All I'm pointing out is that if he's using ssh as a transport, his throughput is limited by ssh itself (specifically the static receive window, unless he's using the HPN ssh patches).
    – JimB
    Jun 10, 2011 at 15:28
1

If the disk subsystem of the receiving server is an array with multiple disks, running multiple rsync processes can improve performance. I am running 3 rsync processes to copy files to an NFS server (RAID6 with 6 disks per raid group) to saturate Gigabit Ethernet.

This guy reports on a basic python harness that spawns multiple rsync processes http://www.reliam.com/company/featured_geek

1
  • Unfortunately, the link is dead. Could you find it again?
    – P.Péter
    Sep 27, 2018 at 7:53
1

I've read many questions similar to this. I think the only real answer is break up the copy/move manually. IOps will be the issue here. If it makes you feel any better, I'm in the process of moving ~200 milllion files consuming well over 100TB of disk space.

0

You may consider checking out the multithreaded cp clone for linux (open source): http://static.usenix.org/event/lisa10/tech/slides/kolano.pdf

1
  • Whilst this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference.
    – Scott Pack
    Oct 19, 2012 at 22:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .