2

Have an Ubuntu machine with a huge collection of files (2.7TB, thousands of directories, over just under 2 million files). I need a daily incremental file backup that a user can easily browse the backup just like they were browsing any other file directory (via Windows Explorer and mapped SMB drive).

Until the backup got so large, a combination of using cp -al to rotate the most recent backup target folder to the previous day using hardlinks and then doing an rsync to the most recent backup target folder was working just great. However I moved the script off the NAS (so it was 'pulling' the files to backup) to a new server and now I'm running the backup script on the source server instead of the backup target device.

I'm not sure if switching from a pull to push is causing the issue or if the fileset is just too large but I'm the script is failing and there is no error ouput from the cp or rsync with detailed logging. It just stops, and I'm finding cp and rsync processes still running in memory that don't appear to be doing anything. It is as if rsync is 'crashing' but not fully removing itself from memory.

The source date, about 95% or more of it does not change as it's archival data but it CAN change once in a while. So the obvious solution would be to segment the backup to only do the most recent directories and then backup the fairly static directories separately at a less frequent interval. OR, change to a completely different backup solution.

But as I said, constraints are that the backup must be easily browsable in Windows Explorer via a mapped drive.

So I'm wondering if there's any rsync option (or other trick) I could use to speed up the backup? It's almost like what I need is for rsync to be able to tell if any files have been changed in a directory without having to read file info for every file and THEN dive into the directory.

I'm using rsync with options: -rlptgoh (oops, I just noticed I took the v for verbose out at some point. Well I'll go ahead and add that back in and see if I get additional info on the problem)

But still interested if there are any suggestions on a better approach given the requirements, or some other combination of rsync options? The ability to browse the incrementals via Windows Explorer really throws a curve ball at Sys Admins' typical recommendations as they don't usually meet that requirement.

1
  • If you could switch to something like zfs on the source/target, this would become far easier with snapshots and zfs send. The sanoid+syncoid zfs tools make this really easy.
    – Zoredache
    May 16, 2019 at 19:57

1 Answer 1

1

To speed up rsync itself you can try using the --numeric-ids option. Moreover, as rsync heavily depends on metadata access speed, you can try setting vfs_cache_pressure=20 on your backup destination.

However from what I understand you are facing a blocked cp or rsync process, which is an entirely different can of worms. I would try to simplify the process, by returning to a pull backup model, which will enable you to use rsync integrated hardlink feature, called --link-dest

Even better, I would use rsnapshot to configure and automate the backup and rotation process. I am using this system for backing up 7+ TB from various server, actually (and I have the same requirement as you: to present the backups via a read-only samba share).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .