Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a customer that upload files through FTP, I made a script that basically monitor the ftp folder and move the file on a production machine. How can I check if the file has been completely transferred from the customer through ftp before start my scp command in order to avoid to transfer partial files?

share|improve this question
add comment

3 Answers

up vote 3 down vote accepted

If your OS supports it you can take a look at incrond. The events that can be monitored are defined in the incrontab Perhaps the event your looking for is

IN_CLOSE_WRITE File opened for writing was closed (*)

or in combination with

IN_CREATE File/directory created in watched directory (*)

Note that

When monitoring a directory, ... the events above can occur for files in the directory, in which case the name field in the returned event data identifies the name of the file within the directory.

share|improve this answer
    
This seems to be very cool, nice didn't know it –  0wn3r Feb 24 '13 at 15:40
add comment

You're not going to be able to do this correctly from bash without help from your customer.

On the FTP server side, several FTP servers have methods for running a program once an upload has terminated (it could have been finished or the client could have been disconnected, the FTP server has no way of knowing if it's really done or not).

To do it right, the best solution is to have the client upload the file with a temporary name, then rename the file once it's finished to the correct name. Then you exclude all the files named "whatever.tmp" from processing. Otherwise, have the client upload a zero byte lock file, upload the real file, then delete the lock file. You can then check for the existence of the lock file

filelist = ...
if [ -e /home/ftp/incoming/lockfile ]; then exit; fi
... process $filelist ...

If the client uploads a lot, then the lockfile may be there every time you run your script and nothing will get done. In that case, you'll have to check the timestamps of the files against the lockfile. Files older than the lockfile should be complete:

for f in ...; do
    if [ ! -e /home/ftp/incoming/lockfile -o $f -ot /home/ftp/incoming/lockfile ]; then
         ... process $f
    fi
done

If neither your FTP server nor your client will participate with you, you can do something like

  1. create a list of files and filesizes.
  2. Compare this list to the previous list of files
  3. process the files on the current list that are identical to the files on the previous list
  4. save the current list of files over the previous list
  5. wait 5 minutes then go back to 1.

Files that haven't gotten bigger in 5 minutes are probably as done as they're going to get. Doing this in bash is left as an exercise to the reader.

share|improve this answer
add comment

The solution I use (which admittedly isn't bulletproof, but has worked for me 100% of the time so far) is to query the file size every 10 seconds, and if it hasn't changed since the last query, assume it is done and begin the transfer.

Granted, it's possible the network could have a 10+ second blip whereby it hasn't finished transferring everything, but that is a pretty rare occurrence, and that is what checksums such as MD5 are for.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.