4

We have an Apache HTTP server up and running, which serves dynamically created zip archives to the users which are possibly several 100s of megabytes in size. As we create a new file with every click on the "download" button (even if the content did not change...), we will most likely run into disc capacity problems.

I have a rather dumb script running that deletes all files that have been created 90+minutes ago. Obviously not that nice of a solution.

I was wondering how to determine from the command line of the server when a download finished, successfully or not. In that case I can delete that file as it will not be served twice. Thinking about it, it might even be sufficient to check if the server is using a file or not as it is created right before it is served to the client.

Thanks!

7 Answers 7

10

Since deleting a file that a process has an open file handle on leaves the file there until the handle is closed you could just delete them immediately, and when Apache closes the file it'll be removed from disk.

3

TRS-80's got the right idea, and I'd definitely recommend going down that route. If you're dead-set on waiting until the transfer's finished, then consider using lsof to determine when nobody's got the file open. So, something like:

for file in /directory/full/of/zips/*.zip; do
  if [ -z "$(lsof $file)" ]; then
    # Nobody's reading it, delete
    rm $file
  fi
done
2

The first way I see is to parse the result of server-status to know if the download has started. When the download has started you can delete the file as TRS-80 said. But I would not recommend that because if you client got disconnect for any reason he will not be possible to restart the download. So I would parse logs file to know when the file has been downloaded. The entry in the log file will not be added untill the connexion with the client is closed. In the log file you will have the number of octet served to the client so that you can compare with the file size to be sure that he has download the whole file.

1

I would propose a more elegant solution:

Apache is capable of conditional logging, and the log can be output to a process. So you can do something like:

SetEnvIf Request_URI "^/path/to/files/.*\.zip$" deletefile
CustomLog "|/path/to/program" "%r" env=deletefile

The program will get the filename after each request is finished and can delete it:

#!/usr/bin/perl
$| = 1;
while (<STDIN>) {
    unlink($_);
}

You could even use "%>s %r" as a format and delete only if the status is 200.

0

This isn't a definitive answer, but how I'd first think about tackling this.

I'd run a script every hour. This script would enumerate all the file names in the Zip source folder. I'd then get the script to read through the Apache logs for some kind of Transfer Completion entry matching the current filename. If there is a matching log entry, delete the file. If not, then move onto the next filename.

0

There is some really great info on this page. I don't feel worth to even contribute due to the cleverness of TRS-80's approach. What concerns me though is that you are serving huge dynamically generated files, but are worried about disk space. I want to make sure you are being wise with your most precious resource, RAM.

First, you must make sure that you are doing things in such a way that Apache can utilize sendfile. I would also be concerned about generating the file with any module based application layer, mod_php, mod_python, or reverse proxy to mongrel/Ruby on Rails. You really need be cautious of this. I don't know much about your setup, but instinct tells me you should:

  1. Use worker MPM instead of Pre-Fork
  2. If using Python see: WSGI If using PHP see: FastCGI If Rails see: Passenger
  3. Don't let the users request which triggers the file generation - do the delivery. Use an AJAX-like pattern to:
    1. Queue generation of the file
    2. Check periodically for completion
    3. Log that downloading has begun (well, it's about to)
    4. Begin downloading
  4. However, do not rely on the client to indicate to you that you should remove the file. I would periodically use lsof on the the "begun log" for removal.

Of course, in my industry we always have to worry about being able to scale. You may not care.

0

As another option -- I have a similar process, but I don't write anything to disk, as I serve multi-GB archives.

Instead, I just emit the appropriate HTTP headers (including Content-Disposition to set the filename), and then hand off to zip (or tar) with the appropriate flags for them to write to stdout.

As for scaling -- I have large files, but don't send them that often. I do pass through 'nice' so I can drop the archival process's priority.

My only concern with my system is the inability to recover a partial transfer without starting over, but you've specifically said you want to clean up both successful and unsuccessful transfers.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .