I have a folder with about 1.5 million files and I need them to be copied to a secondary server as often as possible. What would you recommend because clearly standard rsync commands do not make it because just sending the incremental file list (while using rsync -Pcr options) takes about 30 minutes... and the file transfer about 10 seconds. How would you solve this issue fellow SysAdmins?
I currently use the following command with a 30 minute interval:
rsync -Pcr /var/primary/storage /var/secondary/ --log-file=/tmp/rsync.log
Note: The /var/primary/storage is a folder mounted to the secondary server via NFS
I've also tried now the following command:
cp -aur /var/primary/storage /var/secondary/
But it is odd for me that the command asks me if I allow the overwriting of a destination file that has the exact same modified date as the source file. Isn't the update option supposed to watch it and copy by default if a source file is newer (or modified) than the destination file?