Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've got several TB and many thousands of files I need to move across the network from one Linux host to another. I'd like to get suggestions on how to script this with the following points in mind:

  • files to be moved will fit a date profile (ie. "older than" x years)
  • failure of one file should not prevent the rest from being moved (continue_on_error) and should be noted in a log file
  • files on source host should be automatically deleted after being moved successfully
  • the directory structure on the source host contains many files that are NOT to be moved because they don't fit the date profile. these files should not be touched
  • script should run continuously without user intervention until finished
  • whether files are moved sequentially or in parallel is unimportant

xargs and Rsync would be fine except it does not seem to provide the ability to "delete" the source file after it has been successfully copied.

share|improve this question

closed as off-topic by Nathan C, mdpc, Falcon Momot, Bryan, Ward Jul 25 '13 at 5:05

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions about software development are off-topic here, but can be asked on Stack Overflow." – Nathan C, mdpc, Ward
If this question can be reworded to fit the rules in the help center, please edit the question.

Rsync does have an option for removing the source files. Combined with the --checksum flag it should be trustworthy enough for this kind of operation.

However, you could use git-annex for the migration (it will use rsync under the hood for the actual transfers).

On host A:

mkdir annex
git init
git annex init "host A"

On host B:

git clone ssh://A/path/to/annex annex
git annex init "host B"
git annex sync

Once again on host A:

git remote add b ssh://B/path/to/annex

Add the files to git-annex on host A, after copying them to the annex folder:

git annex add .
git commit . -m "Added files"
find /path/to/files* -mtime +5 -exec git annex move {} --to b\;
git-annex sync

See the git-annex walkthrough for details on usage.

share|improve this answer
Excellent suggestions. I will experiment with these and make a decision on which approach to take. Thanks! – 80skeys Jul 24 '13 at 23:03
The rsync --remove-source-files worked like a charm. I'm not sure how I missed it in the man pages. Thanks for calling my attention to it! – 80skeys Jul 25 '13 at 17:18
Rsync has a quite extensive man page :) Consider marking the response as "accepted" (useful for other users searching for the same type of question). – Niklas Lindblad Jul 25 '13 at 18:46

Not the answer you're looking for? Browse other questions tagged or ask your own question.