I need to migrate about 1TB of data comprised of smaller files (most under 100KB) to another server. I've not even completely enumerated the files but estimates are between 1-2 million.
The initial copy using SCP took over a week. Now we have to synchronize changes. Hundreds to thousands of files are added daily.
I've attemped using rsync (v3) but it is taking too long. By the time it finishes, we will be back to having data out of sync again.
I've seen similar questions here but they are a bit older and wonder if there are any new tools to help with this process.
Issues are futher complicated by the source data being on a shared iSCSI system with poor read performance.
The latest strategy may be to redo the data migration and have the developers write a tool to log all of the new files that are added during the migration process. The directory structure keys off a unique identifier is is very broad and deep, so new files are scattered within this structure and rewritting the app to put new files into a specific directory will not work.
Any strategies appreciated.
OS is RHEL 5 going to RHEL 6.