2

I've got a script that continuously monitors an FTP dropbox directory and processes new files when they come in. The script runs lsof on each file before it tries to process it to determine if the file is still being transferred (some files are large and take a while to transfer) to avoid processing an incomplete file.

Everything has been working fine, except with this one specific FTP client. For some reason, when this client uploads a file to the FTP server, running lsof on the file returns nothing. I tested this by running watch -n0.1 'lsof /path/to/file.txt' and then uploading 'file.txt' to the server using two different FTP clients. When uploading with Cyberduck, lsof returns data about the open file, when using the other FTP client, lsof returns nothing. I don't understand how that could be ( and FWIW, I get the same results with the fuser command). I'm also running watch -n0.1 'ls -l /path/to/file' and can see the size of the file changing as the clients are transferring, so I know the clients are writing to the file.

I know there are workarounds to solve my problem, but I'm more interested in exactly how this FTP client can be writing to a file without that file being 'open' as far as the system is concerned. And ideas?

I'm running vsftp on Ubuntu Server 12.04 LTS

Update:

It turns out that when I tested this earlier using Cyberduck, I was connecting to the FTP server using SFTP, not FTP. When I upload a file with Cyberduck using FTP, I run into the same problem--the file uploads, but nothing shows up in lsof. I also used the standard FTP client in Debian to upload a file using PUT and APPEND--again, same thing.

It looks like all FTP uploads (regardless of client) write to their respective file without "opening" the file as far as the system is concerned. I tested this scenario on another FTP server, and everything works as expected--when the files are being uploaded, they show as open in lsof. So, there is something wrong with the specific configuration of my FTP server. I suspect it has something to do with the version of lsof and/or vsftpd. They are older versions, 4.81 and 2.3.2 respectively.

Improve this question
2
  • What is the "faulty" client? Isn't it writing to a temp file and the renaming it after transfer completes?
    – tonioc
    Jul 4, 2014 at 11:47
  • The FTP client is a built-in part of my company's ERP system. I don't think it's writing to a temp file first as I can see the file size grow steadily as the file is being transferred. Also, the only file that's returned by ls is the file in question.
    – bcst
    Jul 4, 2014 at 12:19

1 Answer 1

Reset to default
0

In addition to the STOR (PUT) most of us know and would use, a relatively uncommon FTP command to upload data to a remote server is APPEND :

     APPEND (with create) (APPE)

        This command causes the server-DTP to accept the data
        transferred via the data connection and to store the data in
        a file at the server site.  If the file specified in the
        pathname exists at the server site, then the data shall be
        appended to that file; otherwise the file specified in the
        pathname shall be created at the server site.

The could allow a client to add to a file without keeping it open the whole time...

Improve this answer
2
  • Wow, I never knew about APPEND. That would make perfect sense. I'll test out APPEND with another FTP client as see if it behaves the same way. I'll report back the results. Thanks!
    – bcst
    Jul 4, 2014 at 18:04
  • I tested the APPEND and PUT commands using the Debian ftp client, and I get the same results regardless of the command used to upload.
    – bcst
    Jul 7, 2014 at 14:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .