3

I'm using rsync to transfer application log files from one server to another. It works fine with the exception that scanning the directories takes forever. I have around 160,000 log files and a few thousand directories so I can understand why scanning takes so long. However, only a few of those files changes at any given sync so I'm wondering if there's a more efficient way or rsync option that may speed this up? The log files themselves are tiny (only a few kb each).

Thank you!

1
  • The simple fact is that any method of copying 160,000 files is going to be slow, other then a sector-by-sector copy of the filesystem. If this is something you need to do often, then you should probably look at some of the options for distributed filesystems.
    – Zoredache
    Feb 21, 2013 at 23:21

1 Answer 1

2

From the manpage:

Beginning with rsync 3.0.0, the recursive algorithm used is now an incremental scan that uses much less memory than before and begins the transfer after the scanning of the first few directories have been completed. This incremental scan only affects our recursion algorithm, and does not change a non-recursive transfer. It is also only possible when both ends of the transfer are at least version 3.0.0.

Some options require rsync to know the full file list, so these options disable the incremental recursion mode. These include: --delete-before, --delete-after, --prune-empty-dirs, and --delay-updates. Because of this, the default delete mode when you specify --delete is now --delete-during when both ends of the connection are at least 3.0.0 (use --del or --delete-during to request this improved deletion mode explicitly). See also the --delete-delay option that is a better choice than using --delete-after.

If you're using any of those mentioned options or older versions of rsync you may be disabling the improved incremental recursion mode. Otherwise the only other option is to give rsync fewer files to consider.

1
  • Thanks for your help! I'm at 3.0.6 and not using any of those options. I just decided to exclude previous years from the sync. Only thing that sucks is I have to remember to update the cronjob every year to add an exclusion for the past year. Appreciate your feedback!
    – Jason
    Feb 21, 2013 at 23:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .