Have an Ubuntu machine with a huge collection of files (2.7TB, thousands of directories, over just under 2 million files). I need a daily incremental file backup that a user can easily browse the backup just like they were browsing any other file directory (via Windows Explorer and mapped SMB drive).
Until the backup got so large, a combination of using cp -al to rotate the most recent backup target folder to the previous day using hardlinks and then doing an rsync to the most recent backup target folder was working just great. However I moved the script off the NAS (so it was 'pulling' the files to backup) to a new server and now I'm running the backup script on the source server instead of the backup target device.
I'm not sure if switching from a pull to push is causing the issue or if the fileset is just too large but I'm the script is failing and there is no error ouput from the cp or rsync with detailed logging. It just stops, and I'm finding cp and rsync processes still running in memory that don't appear to be doing anything. It is as if rsync is 'crashing' but not fully removing itself from memory.
The source date, about 95% or more of it does not change as it's archival data but it CAN change once in a while. So the obvious solution would be to segment the backup to only do the most recent directories and then backup the fairly static directories separately at a less frequent interval. OR, change to a completely different backup solution.
But as I said, constraints are that the backup must be easily browsable in Windows Explorer via a mapped drive.
So I'm wondering if there's any rsync option (or other trick) I could use to speed up the backup? It's almost like what I need is for rsync to be able to tell if any files have been changed in a directory without having to read file info for every file and THEN dive into the directory.
I'm using rsync with options: -rlptgoh (oops, I just noticed I took the v for verbose out at some point. Well I'll go ahead and add that back in and see if I get additional info on the problem)
But still interested if there are any suggestions on a better approach given the requirements, or some other combination of rsync options? The ability to browse the incrementals via Windows Explorer really throws a curve ball at Sys Admins' typical recommendations as they don't usually meet that requirement.