I have a single root directory (on a NAS) that I need to backup to another server. The hierarchy of the directory is several layers deep and a majority of the data is composed of extremely large files collections inside about a dozen directories located in various locations throughout the hierarchy. If I do an rsync on the parent directory, I can get a throughput of about 100MB/s. I've tested running multiple rsync processes manually on different directories and can get a higher throughput (350-500MB/s). I'm running rsync from the destination server.
I've implemented a naive solution, where it just splits up the rsyncs using the directories directly under the parent directory, but it needs to be more targeted than that, since some of those directories contain more data than others. Ideally, each process would be responsible for syncing roughly the same amount of data.
Is there a way to use rsync so that it splits up a task into n rsync processes, each with a (roughly) equal number of files.