We have a large amount of files on a remote server that I'd like to setup regular backups to a local system for extra redundancy. Some details:
- Remote system is not in my control. I only have SSH/rsync or FTP access
- Remote system runs rsync 2.6.6 and cannot be upgraded
- Remote system allows a max of 25 concurrent connections and 5 are reserved for production needs (so, 20 available)
- Remote system contains 2M files - the majority of which are 100-200K in size
- Files are stored in a hierarchy
0123456789/ 0123456 abc/ 1.fff 2.fff 3.fff xyz/ 9.fff 8.fff 7.fff 9877656578/ 5674563 abc/ 1.fff 2.fff 3.fff xyz/ 9.fff 8.fff 7.fff
with 10's of thousands of those root folders containing just a few of the internal folder/file structures - but all root folders are numeric (0-9) only.
I ran this with a straight
rsync -aP the first time and it took
3196m20.040s. This is partially due to the fact that since the remote server is on
rsync 2.6.6 I can't use the incremental file features found in 3.x.x. It takes almost 12 hours to compile the file list - running about 500 files per 10 seconds. I don't anticipate subsequent runs will take as long because the initial run had to download everything anew - however even 12 hours just for the file listing is too long.
The folder naming is broken up as such:
$ ls | grep "^[^67]" | wc -l 295 $ ls | grep "^6" | wc -l 14167 $ ls | grep "^7" | wc -l 14414
I've tested running this
rsync -aWP --delete-during by breaking it up using
--include="/0*/" --exclude="/*/" where I run 8 of these concurrently with
0* 1* 2* 3* 4* 5* 8* 9* and for 6 and 7 I use
70*-79* because the brunt of the folders in the hierarchy begin with
7 (roughly 1400 per
Everything that's not a 6 or 7 takes about 5 minutes, total. The 6/7 directories (broken down in 1/10ths) take 15 minutes each.
This is quite performant, except to run this job I have to run 28 concurrent
rsync and this saturates the available connection count - not to mention potentially saturating the network.
Does anyone have a recommendation for another variant of
rsync or some additional options I could add to prevent this from using so many connections concurrently without having to stage this sequentially in the bounds of
rsync 2.6.6 on one end?
Edit #1: We do pay for bandwidth to/from this external provider so ideally we would only send things over the wire that need to be sent, and nothing more.