We have a large amount of files on a remote server that I'd like to setup regular backups to a local system for extra redundancy. Some details:

  • Remote system is not in my control. I only have SSH/rsync or FTP access
  • Remote system runs rsync 2.6.6 and cannot be upgraded
  • Remote system allows a max of 25 concurrent connections and 5 are reserved for production needs (so, 20 available)
  • Remote system contains 2M files - the majority of which are 100-200K in size
  • Files are stored in a hierarchy

Similar to:


with 10's of thousands of those root folders containing just a few of the internal folder/file structures - but all root folders are numeric (0-9) only.

I ran this with a straight rsync -aP the first time and it took 3196m20.040s. This is partially due to the fact that since the remote server is on rsync 2.6.6 I can't use the incremental file features found in 3.x.x. It takes almost 12 hours to compile the file list - running about 500 files per 10 seconds. I don't anticipate subsequent runs will take as long because the initial run had to download everything anew - however even 12 hours just for the file listing is too long.

The folder naming is broken up as such:

$ ls | grep "^[^67]" | wc -l
$ ls | grep "^6" | wc -l
$ ls | grep "^7" | wc -l

I've tested running this rsync -aWP --delete-during by breaking it up using --include="/0*/" --exclude="/*/" where I run 8 of these concurrently with 0* 1* 2* 3* 4* 5* 8* 9* and for 6 and 7 I use 60*-69* and 70*-79* because the brunt of the folders in the hierarchy begin with 6 or 7 (roughly 1400 per 6?* or 7?*).

Everything that's not a 6 or 7 takes about 5 minutes, total. The 6/7 directories (broken down in 1/10ths) take 15 minutes each.

This is quite performant, except to run this job I have to run 28 concurrent rsync and this saturates the available connection count - not to mention potentially saturating the network.

Does anyone have a recommendation for another variant of rsync or some additional options I could add to prevent this from using so many connections concurrently without having to stage this sequentially in the bounds of rsync 2.6.6 on one end?

Edit #1: We do pay for bandwidth to/from this external provider so ideally we would only send things over the wire that need to be sent, and nothing more.

  • 1
    What is your recovery point objective? If last week as the point in time is acceptable, so is the backup taking 3 days. Jan 22 '19 at 19:47
  • daily for 7 days, weeklies for a month, monthlies for 3 months, annuals beyond that. I'd like <24 hours on the backups in case of an issue.
    – jimmy0x52
    Jan 22 '19 at 19:49
  • How often do existing files change? How often are new files created? IE: Are the differences mostly updated files or new files? Not that I have a brilliant solution, but this information may make one approach better. Jan 22 '19 at 20:15
  • Existing files change rarely - if ever. It's mostly adding new root or 2nd level numbered folders, with unique files contained inside of that 2nd level.
    – jimmy0x52
    Jan 22 '19 at 20:17
  • 1
    I've been doing my homework :) -- I'll give it a try and let you know. I'm also going to try to re-run this job a second time, now that the data is actually copied down -- and get a sense of how long a second pass takes. I will post results (it might take a couple days)
    – jimmy0x52
    Jan 23 '19 at 14:36

After an initial sync time of 40 hours to download and sync all of the data a subsequent scan and sync of the same data (just to pull in updates) only took 6.5 hours. The command used to run the rsync was:


I think my large initial time for download was twofold:

  1. The initial dataset is 270GB and ~2M files, which is a lot to scan and download over the internet (in our case we have a 100mbit synchronous connection and this was connecting to a large CDN provider)

  2. I had the -P option enabled and -v options on the initial sync which caused a lot of local console chatter displaying every file being synced and progress information.

So, the answer here: Just use rsync with not so many verbosity options (and --quiet ideally) and it's quite efficient - even to huge datasets.

  • 1
    Pleased to see it's working as it should. If some of your data is compressible, consider adding -z.
    – roaima
    Jan 24 '19 at 18:24
  • 1
    Being as efficient as possible on these kinds of datasets is really the whole point of rsync. I run a server which does over 100 such rsync calls a day to 30 different remote servers. The initial sync to some of these took several days, but the later incremental updates are not unreasonably long. One extreme example is 1.3TB in 3.4M files, which took almost a week for its initial sync and now takes 4-10 hours a day to check for updates.
    – Moshe Katz
    Jan 25 '19 at 2:16
  • 1
    Given the low file sizes and high count, you might (I stress might) see some improvement with -W in your rsync invocation. It just copies whole files based on size/timestamp changes and skips any delta transmission. Depending on the nature of the changes it can save time. Feb 3 '19 at 13:00

Here's what I would personally do - there are two variations to the solution.

Variation 1 - the simple, bruteforce option:

2M * 200KB is roughly 400GB, so a full snapshot every time may not be possible. If it is possible, the simple solution would be to perform:

ssh <remote host> 'tar -c /directory/to/backup | <gzip/xz/lz4>' > backup.tar.<gz/xz/lz4>

How this works is turning all of those files into a single stream that's pushed across the pipe rather than Rsync/SFTP enumerating the millions of files.

From there, I would use borg to deduplicate the tar ball so you can efficiently store multiple versions. This is a common trick for piping tons of small files very fast. The downside is that you can't do the deduplication that RSync performs.

If the 400GB per interval is too large, I'd consider the following:

Variation 2 - the clever option.

You could perform the following, except you would create a tarball for each top level directory and compare the hash to the existing file on the backup server. If it's different, transfer it, otherwise, don't do anything.

  • 400GB over the wire, every night, might be pushing it. I didn't mention in the initial post but we do pay for bandwidth from this provider so anything I can do to reduce the transfer of items that don't need to be transferred, the better.
    – jimmy0x52
    Jan 22 '19 at 21:06
  • 1
    Do the files compress well? Jan 22 '19 at 21:39
  • Good question - I am tarring them locally and I'll let you know. I did look and uncompressed it's about 270GB (so not all of the 2M files are of the 100-200k size). They're about half binary half text also, so we shall see.
    – jimmy0x52
    Jan 22 '19 at 21:50
  • 270GB -> 144GB after tar/gzipping. Not bad. I will explore this option further. Thanks.
    – jimmy0x52
    Jan 23 '19 at 13:37

2M files means a lot of metadata, so your rsync runs are slow due to both the local and remote side needing to walk all of the metadata.

You should maximize RAM at both ends, and ideally run with rsync version > 3.x The fact that you can't update rsync at the remote end let me think you can not update RAM, though.

A last attempt would be to prioritize, on both local and remote side, metadata caching. You can try setting vfs_cache_pressure=10, re-run rsync at least two times and compare the second run performance after modifying the parameter above.

  • vfs_cache_pressure assumes Linux, the operating system has not yet been provided. Jan 23 '19 at 12:45
  • Zero control over the hardware/software on the remote side. Using a 3rd party CDN provider for the remote piece.
    – jimmy0x52
    Jan 23 '19 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.