2

I have an sftp server that partners send files to. As soon as the file arrives in the sftp directory, I want to read its contents and send the contents for further processing to another server.

To achieve the above, I have set up a very thin file watcher program running on the same machine as the sftp server. It uses a file system events watcher library to subscribe to CREATE events - whenever such an event is fired, the watcher reads the file and sends its contents to the processing server.

This works locally. That is, if I mv a file to the sftp directory from the same machine, the contents are correctly parsed. However, when I actually put a file into the sftp directory from a remote machine, the following happens:

  1. CREATE fs event triggered
  2. SFTP starts transferring data
  3. File watcher receives CREATE event, opens partially written file, sends partial contents to remote processing server.
  4. SFTP finishes transferring data.

As a result, I end up with empty contents on the remote server since it reads the file before any data has been transferred to it. I have verified that the files eventually receive all data.

What sequence of FS events is triggered by a SFTP put? How should I solve for the above use case? I was exploring simple delays (once you receive CREATE event, wait 5 seconds, then read file) but none seem sustainable.

2
  • 1
    A fairly typical solution is uploading to temporary location or temporary filename followed by a rename/move operation which is atomic. This needs to implemented at the client though. See for example : winscp.net/eng/docs/resume#automatic
    – Bob
    Mar 21 '21 at 19:19
  • @HermanB Thanks. Let's say I do not have control over the clients. Are there any alternatives? Other than simple delays, I was exploring the possibility of delays with exponential backoff, where each iteration resumes waiting if it sees the file currently has one open file descriptor (assuming this can be observed from user space)
    – bkr879
    Mar 21 '21 at 19:25
1

If relying on inotify, you should watch on CLOSE_WRITE event rather than CREATE.

If you don't need recursive monitoring, you can give a look at incrond (and incrontab)

Alternatively, you can simply schedule rsync to run with short interval (ie: 1 min) and clean up the source dir in off-working hours, when you can stop the SFTP service (to be 100% certain no one is uploading a file during the cleaning operation).

EDIT: well, it seems the your library of choice does not provide the CLOSE_WRITE event, but only MODIFY (see here). The issue with MODIFY is that any writes will trigger a separate event, which means a single big file upload can trigger an undefined amount of MODIFY events.

If you want to continue using a notify library, I suggest you to evaluate incrond, lsyncd or inotifywait

Regarding the plain rsync approach, you can surely end transferring a partially uploaded file, which will howevever be completely transferred the next rsync cycle after its initial upload finished. On the receiving side, you should be sure to only process completely transferred files (a thing you should check even with a notify approach).

More broadly, I strongly suggest you to test with a representative file size sample because testing with small files can hide some timing-related issues which affect bigger files.

7
  • It does seem like the above library uses inotify under the hood. Interestingly, however, this is the granularity of events it provides: Create, Write, Remove, Rename, Chmod (where each is a bit mask, so they can be combined). So, I believe the CLOSE_WRITE event would not be covered. I am wondering about 1 min interval solution - what if it happens to be triggered while the file is being written to?
    – bkr879
    Mar 21 '21 at 19:28
  • After more experimentation, I see that each sftp put first triggers a CREATE and then a WRITE (no other events are generated - only those two). For a small file, they are a few milliseconds apart. Perhaps that means the WRITE happens after the file has been written to...
    – bkr879
    Mar 21 '21 at 19:34
  • I just validated the above assumption (at least on small files) - in that specific library the WRITE event seems to be fired after the file has been fully written to (probably on close of the file by the sftp connection). I will monitor to validate this on larger files (which would potentially fail if it turns out the WRITE event is triggered before the file has been fully written to) and update here.
    – bkr879
    Mar 21 '21 at 19:44
  • @bkr879 I've edited my answer, give it a look.
    – shodanshok
    Mar 21 '21 at 20:20
  • Thank you. Is there a reliable way to estimate the number of WRITE events issued as a function of file size? So far all the files I have tested have triggered only one WRITE (max was 55KB). I suppose I can test it out myself with an arbitrarily large file, but my concern is that this also depends on the sftp client logic (perhaps it somehow splits up large files and sends each chunk separately)
    – bkr879
    Mar 21 '21 at 23:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.