Bash script: Monitor hotfolder and partial files

Question

I have a customer that upload files through FTP, I made a script that basically monitor the ftp folder and move the file on a production machine. How can I check if the file has been completely transferred from the customer through ftp before start my scp command in order to avoid to transfer partial files?

Answer 1

If your OS supports it you can take a look at incrond. The events that can be monitored are defined in the incrontab Perhaps the event your looking for is

IN_CLOSE_WRITE File opened for writing was closed (*)

or in combination with

IN_CREATE File/directory created in watched directory (*)

Note that

When monitoring a directory, ... the events above can occur for files in the directory, in which case the name field in the returned event data identifies the name of the file within the directory.

Answer 2

You're not going to be able to do this correctly from bash without help from your customer.

On the FTP server side, several FTP servers have methods for running a program once an upload has terminated (it could have been finished or the client could have been disconnected, the FTP server has no way of knowing if it's really done or not).

To do it right, the best solution is to have the client upload the file with a temporary name, then rename the file once it's finished to the correct name. Then you exclude all the files named "whatever.tmp" from processing. Otherwise, have the client upload a zero byte lock file, upload the real file, then delete the lock file. You can then check for the existence of the lock file

filelist = ...
if [ -e /home/ftp/incoming/lockfile ]; then exit; fi
... process $filelist ...

If the client uploads a lot, then the lockfile may be there every time you run your script and nothing will get done. In that case, you'll have to check the timestamps of the files against the lockfile. Files older than the lockfile should be complete:

for f in ...; do
    if [ ! -e /home/ftp/incoming/lockfile -o $f -ot /home/ftp/incoming/lockfile ]; then
         ... process $f
    fi
done

If neither your FTP server nor your client will participate with you, you can do something like

1. create a list of files and filesizes.
2. Compare this list to the previous list of files
3. process the files on the current list that are identical to the files on the previous list
4. save the current list of files over the previous list
5. wait 5 minutes then go back to 1.

Files that haven't gotten bigger in 5 minutes are probably as done as they're going to get. Doing this in bash is left as an exercise to the reader.

Answer 3

The solution I use (which admittedly isn't bulletproof, but has worked for me 100% of the time so far) is to query the file size every 10 seconds, and if it hasn't changed since the last query, assume it is done and begin the transfer.

Granted, it's possible the network could have a 10+ second blip whereby it hasn't finished transferring everything, but that is a pretty rare occurrence, and that is what checksums such as MD5 are for.