Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm working on a system that archives large collections of timetstamped images. Part of the system deals with saving an image to a growing .zip file. This morning I noticed that the log system said that an image was successfully downloaded and placed in the zip file, but when I downloaded the .zip (from an apache alias running on our server), the images didn't match the log. For example, although the log said that camera 3484 captured on January 17, 2011, when I download from the apache alias, the downloaded zip file only contains images up to January 14.

So, I sshed onto the server, and unzipped the file in its own directory, and that zip file has images from January 14 to today (January 17). What strikes me as odd is that this should be the exact same file as the one I downloaded from the apache alias.

Other experiments: I scp-ed the file from the server to my local machine, and the zip file has the newer images. But when I use an SCP client (in this case, Fugu for OSX), I get the zip file for the older images.

In short: unzipping a file on the server or after downloading through scp or after downloading through wget gives one zip file, but unzipping a file from Chrome, Firefox, or SCP client gives a different zip file, when they should be exactly the same.

Unzipping on the server...

[user@server ~]$ cd /export1/amos/images/2011/84/3484/00003484/
[user@server 00003484]$  ls -la
total 6180
drwxr-sr-x 2 user groupname      24 Jan 17 11:20 .
drwxr-sr-x 4 user groupname      36 Jan 11 19:58 ..
-rw-r--r-- 1 user groupname 6309980 Jan 17 12:05 2011.01.zip
[user@server 00003484]$ unzip 2011.01.zip
Archive:  2011.01.zip
extracting: 20110114_140547.jpg     
extracting: 20110114_143554.jpg     
replace 20110114_143554.jpg? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
extracting: 20110114_143554.jpg     
extracting: 20110114_153458.jpg     
   (...bunch of files...)
extracting: 20110117_170459.jpg     
extracting: 20110117_173458.jpg     
extracting: 20110117_180501.jpg 

Using the wget through apache alias.

local:~ user$ wget http://example.com/zipfiles/2011/84/3484/00003484/2011.01.zip
--12:38:13--  http://example.com/zipfiles/2011/84/3484/00003484/2011.01.zip
       => `2011.01.zip'
Resolving example.com... ip.ip.ip.ip
Connecting to example.com|ip.ip.ip.ip|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6,327,747 (6.0M) [application/zip]

100%    [=====================================================================================================>] 6,327,747      1.03M/s    ETA 00:00

12:38:56 (143.23 KB/s) - `2011.01.zip' saved [6327747/6327747]

local:~ user$ unzip 2011.01.zip
Archive:  2011.01.zip
extracting: 20110114_140547.jpg     
(... same as before...)  
extracting: 20110117_183459.jpg 

Using scp to grab the zip

local:~ user$ scp user@server:/export1/amos/images/2011/84/3484/00003484/2011.01.zip .
2011.01.zip                                                                                                    100% 6179KB 475.3KB/s       00:13    
local:~ user$ unzip 2011.01.zip
Archive:  2011.01.zip
extracting: 20110114_140547.jpg     
   (...same as before...)
extracting: 20110117_183459.jpg

Using Fugu to download 2011.01.zip from /export1/amos/images/2011/84/3484/00003484/ gives images 20110113_090457.jpg through 201100114_010554.jpg

Using Firefox to download 2011.01.zip from http://example.com/zipfiles/2011/84/3484/00003484/2011.01.zip gives images 20110113_090457.jpg through 201100114_010554.jpg

Using Chrome gives same results as Firefox.

Relevant section from apache httpd.conf:

# ScriptAlias: This controls which directories contain server scripts.
# ScriptAliases are essentially the same as Aliases, except that
# documents in the realname directory are treated as applications and
# run by the server when requested rather than as documents sent to the client.
# The same rules about trailing "/" apply to ScriptAlias directives as to
# Alias.
#
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
Alias /zipfiles/ /export1/amos/images/
share|improve this question

migrated from stackoverflow.com Jan 17 '11 at 22:29

This question came from our site for professional and enthusiast programmers.

    
We've got a apache server on the lan here, serving multi-gigabyte VM images. We've seen issues with truncated downloads -- the resulting file is frequently hundreds of megs shy of what it should be, and md5sums don't match (naturally). So, it's certainly not hard to get an incomplete download from apache. Why it happens, I'm not sure -- in our case, it could very well have been someone restarting httpd. –  Frank Farmer Jan 17 '11 at 19:08
    
Google turns up a couple of similar hits on the SO network, although the answers aren't great: stackoverflow.com/questions/3320422/… serverfault.com/questions/68560/… ... Is apache running on port 80 without any sort of load balancers/proxies in front of it? –  Frank Farmer Jan 17 '11 at 19:13
    
As far as I can tell, the downloads are all completing. These zip files aren't that huge; it's usually on the order of 25-50 MB, although the one I provided in the example above is just 6 MB. I don't think that it's truncating the results, either, because even though I've seen two distinct files coming from the (supposedly) same location, they've always been the same two files. –  Austin A. Jan 17 '11 at 19:59
    
I tried the results on another local machine and got the same results, so I don't believe this is a local cache issue. –  Austin A. Jan 17 '11 at 20:11
    
try servfault, really, and check the server-status page to see if some process do not have graceful restart problems (do a real stop/start if you can), give your /etc/mtab and mount status (filesystem mounted several timee in same point?). redo the wget example with header trace stackoverflow.com/questions/302195/… and catch headers in Firefox with live http headers to get cache control headers. –  regilero Jan 17 '11 at 22:14

2 Answers 2

You mentionde something that implied the zip file is modified while it's being served.

You can't reliably serve files that are being grown or truncated during the course of the request. The ideal way to shorten this window significantly is to always work on a copy, edit, rm the old file then mv the new one into place (processes that have the old one open continue to serve it, someone who checks "during" the mv gets a 404 and at least knows to retry, and anyone after gets a permanent view of the new file.

Otherwise, if I read too much into that stmt, try EnableSendfile off.

share|improve this answer

I believe I figured it out, and I was just being an idiot.

Due to an error that occurred last week (which has since been fixed), there was a time when one zip file could be modified by two processes that were both trying to append to the same file. So, I believe that due to some zipfile concurrency issue, when both processes were done, it had the effect of two zip files concatenated together. And, it turns out that different unzipping tools will look at different parts of this Franken-zip monster. So, while I was using unzip on the server or after using wget, that looked at one part of the zip file, and while I was using the default OSX GUI tool to unzip, that looked at another part of the zip file.

Downloading one .zip file and using two separate tools validates this theory.

Sorry that the problem wasn't about apache, as I had originally thought.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.