I have a large (several megabytes) XML file that's updated rather frequently (every 10 minutes or less) and gets a lot of traffic. I'd like to implement some caching to reduce bandwidth and server load. Looking at the Apache documents, I see a dizzying array of configuration options that involve various combinations of mod_expires, mod_headers, and mod_cache (and variants). I end up running in circles and the results aren't what I expect.

I'm comfortable editing the various configuration files if I have some idea what I'm supposed to change. But at the moment I'm poking around in the dark and that's never a comfortable feeling. So, perhaps if I describe what I want, somebody here can take me by the hand and say, "This is what you need to do."

Periodically, this file, call it "stuff.xml" is updated and a new version copied to the directory. The external url would be, for example, http://example.com/stuff.xml. Understand, this part works. Whenever I request the file, I get the expected result. But the file is big and I want to save bandwidth, so first I'd like to implement conditional GET semantics with the If-Modified-Since header. How do I do this? I've enabled mod_headers and mod_expired and added the <FilesMatching> section in my httpd.conf as recommended in countless examples I've seen online, but that didn't change the behavior when made a conditional GET request. I always get a status 200 with the entire document. So how the heck do I implement this?

That'll cut down on neeless transfers. I'd also like to limit the amount of data transferred. Seeing as this is XML, gzipping it should save me 50% or more. My next step would be to somehow gzip the file and, if it's not too difficult, store it in memory. That'll cut down on per-access data transfer, and also reduce disk transfers. So how do I implement this type of caching?

Thanks in advance.

  • I discovered the problem with the If-Modified-Since header. Turns out that if the date you supply is in the future (i.e. after the Web server's current date/time), Apache returns a 200. A bug in my test program was sending the wrong date. – Jim Mischel Jun 7 '10 at 22:24

I've setup caching using mod_expires, within the <VirtualHost> definition

ExpiresActive on
ExpiresByType text/xml "modified plus 5 minutes"   

You can setup compression using mod_deflate

SetOutputFilter DEFLATE
AddOutputFilterByType DEFLATE text/xml 
| improve this answer | |
  • Using mod_expires as you suggest, I indeed get an Expires header when I do the GET: whether or not I set the If-Modified-Since header in the request. The Last-Modified header shows 02 Jun 2010. The date in the Expires header is always the same as what's in the Date header--the current date/time. – Jim Mischel Jun 7 '10 at 22:04

As far as why the the If-Modified-Since doesn't work, I think you should post the relevant parts of your configuration (I Assume you have restarted Apache). You might also want to quickly parse the logs to see if the same IP (possibly same user) gets that XML file multiple times. If it always or mostly new users each time, the client-side cache won't help reduce load on your server.

For the compression, you want to look into mod_deflate and/or mod_gzip. Here is an older comparison of the on Linux Journal.

For the memory caching, I think if it is accessed enough that it is worth caching the OS is probably already caching it. If you don't have IO problems or bad latency on the file, I would trust the OS to do its job.

| improve this answer | |
  • I'll hold on the caching. I think you're right: see if I'm having latency issues first. Thanks for the link to the compression article. – Jim Mischel Jun 7 '10 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.