I have an sftp server that partners send files to. As soon as the file arrives in the sftp directory, I want to read its contents and send the contents for further processing to another server.
To achieve the above, I have set up a very thin file watcher program running on the same machine as the sftp server. It uses a file system events watcher library to subscribe to CREATE events - whenever such an event is fired, the watcher reads the file and sends its contents to the processing server.
This works locally. That is, if I
mv a file to the sftp directory from the same machine, the contents are correctly parsed. However, when I actually
put a file into the sftp directory from a remote machine, the following happens:
- CREATE fs event triggered
- SFTP starts transferring data
- File watcher receives CREATE event, opens partially written file, sends partial contents to remote processing server.
- SFTP finishes transferring data.
As a result, I end up with empty contents on the remote server since it reads the file before any data has been transferred to it. I have verified that the files eventually receive all data.
What sequence of FS events is triggered by a SFTP put? How should I solve for the above use case? I was exploring simple delays (once you receive CREATE event, wait 5 seconds, then read file) but none seem sustainable.