Collaboration partners of our company will upload files (typically a few GB in size) to a directory on one of our external servers.
From that directory, I will move them to our internal network, where they will be ultimately consumed by our analysts.
So far, the upload is solved with a chroot jail and works fine, same for for what happens when the files arrive in our internal network.
The transfer to our internal network, however, is problematic. I simply rsync them with --remove-source-files and a find to delete the empty directories.
Thing is, the cron job polling interval on the directory needs to be low (we'd prefer every minute), while the transfer time is fairly high (our office DSL is slow), and obviously we don't want to start uploading the same file every 5 minutes.
Is there a good solution for this problem? I could move the contents to a temporary directory, and then rsync from there, but I'm feeling like a more elegant solution exists.