I've got several TB and many thousands of files I need to move across the network from one Linux host to another. I'd like to get suggestions on how to script this with the following points in mind:
- files to be moved will fit a date profile (ie. "older than" x years)
- failure of one file should not prevent the rest from being moved (continue_on_error) and should be noted in a log file
- files on source host should be automatically deleted after being moved successfully
- the directory structure on the source host contains many files that are NOT to be moved because they don't fit the date profile. these files should not be touched
- script should run continuously without user intervention until finished
- whether files are moved sequentially or in parallel is unimportant
xargs and Rsync would be fine except it does not seem to provide the ability to "delete" the source file after it has been successfully copied.