11

I have recently run into an issue after switching to Cloudflare, and the solution is to basically stop Cloudflare from caching 404 responses.

In our load-balanced multi-server setup, occasional 404s happen, but they're quickly fixed by rsync (via lsyncd). Before Cloudflare, a re-request to the 404ed file would very quickly become 200 as rsync does its job.

However, since Cloudflare caches all data based on the caching header, and neither apache nor nginx send a no-cache header for 404s, Cloudflare ends up caching the 404 response for a while.

I've been searching for a solution to globally add such a header for 404s in both apache and nginx (globally, for all hosted domains), but so far have come up blank.

Can anyone help?

Thank you.

2
  • Similar question, though only for apache (still unanswered too): serverfault.com/questions/331544/…. May 7, 2015 at 5:10
  • So far, I am pretty sure that you can't override headers returned by default 404 handlers in both apache and nginx (please prove me wrong!). I was able to override the 404 handler and point it to a PHP file that sends such headers in apache, but since nginx doesn't have PHP support on my setup, and setting "expires -1;" on the 404 location didn't seem to actually do anything, I'm still at a loss as to how to do this in nginx. May 7, 2015 at 7:08

5 Answers 5

7

Can't you get by with using an error_page directive, and then handle the location separately with the added header?

e.g. in Nginx:

    server {
      ...
      error_page 404 /404.html;
      location = /404.html {
        root   /usr/share/nginx/html;
        add_header Cache-Control "no-cache" always;
      }
    }
5
  • 1
    There are two issues here, but that's the direction I went with. 1. I didn't want to override the default 404 page and have to create rules in every listen since location is not supported inside http directly. 2. More importantly, your snippet actually doesn't work because add_header only applies to 20X and 30X (nginx.org/en/docs/http/ngx_http_headers_module.html). However, we're in luck, since as of the recently released 1.7.5, you can now add an always modifer which will apply it to all response codes. I had to upgrade nginx but it was a good kick in the butt. It works. May 7, 2015 at 8:25
  • 2
    I updated my answer, you should perhaps add an answer showing what you did to solve the question?
    – jimmiw
    May 7, 2015 at 8:37
  • Looks like omitting root works as well. If that's removed, it is pretty much what I ended up doing for nginx. May 7, 2015 at 19:00
  • For apache, I decided that I actually wanted to use a custom page, so I did this: ErrorDocument 404 /path/to/my/404.php. And then inside the 404.php, I did <?php // do not cache 404 header("Cache-Control: no-cache, must-revalidate"); // HTTP/1.1 header("Expires: Sat, 26 Jul 1997 05:00:00 GMT"); // Date in the past ?> May 7, 2015 at 19:04
  • I'm marking your answer as accepted. If someone wants to provide a more generic apache answer later, or an nginx one that works globally rather than per server, they can get upvoted. May 7, 2015 at 19:05
7

You can do it this way too :

map $status $cache_header {
    default <for_other_codes>;
    404     "no-cache";
}


server {

    [ ... ]

    add_header "Cache-Control" $cache_header always;

}
2
  • Clever. I am probably going to go with the other solution, since it already works and doesn't require a default Cache-Control header to be specified every time, but this was some thinking outside the box. May 7, 2015 at 19:01
  • There is a semicolon missing after 404 "no-cache" but stackexchange's stupid minimum-6-character-edit restriction prevents me from fixing it. Clearly not a good restriction for a site that's all about coding and configs...
    – nh2
    Jul 3, 2017 at 0:48
6

In apache 2.4, you could try something like:

FileETag None
<IfModule mod_headers.c>
    Header always set Cache-Control "max-age=0, no-cache, no-store, must-revalidate" "expr=%{REQUEST_STATUS} == 404"
    Header always set Expires "Wed, 11 Jan 1984 05:00:00 GMT" "expr=%{REQUEST_STATUS} == 404"
</IfModule>

The always is important because this is a:

You're adding a header to a locally generated non-success (non-2xx) response, such as a redirect, in which case only the table corresponding to always is used in the ultimate response.

You said all 404s, but for full reference of course it might make sense to wrap that in a <FilesMatch> or <LocationMatch> to limit the scope.

I believe this is a new capability in apache 2.4 as using expr conditionals is not the in the 2.2 version of the mod_headers documentation.

curl -I [foo] test without this config:

HTTP/1.1 404 Not Found
Date: Thu, 24 May 2018 17:44:29 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1

curl -I [foo] test with this config:

HTTP/1.1 404 Not Found
Date: Thu, 24 May 2018 17:44:42 GMT
Server: Apache/2.4.18 (Ubuntu)
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Content-Type: text/html; charset=iso-8859-1

Sources:

http://httpd.apache.org/docs/current/mod/mod_headers.html

2
  • Removing the Apache generated etag is not going to change the behaviour. Sending "Pragma: no-cache" in a response header is meaningless.
    – symcbean
    Aug 22, 2023 at 9:40
  • Ah, 2018 cayleaf was a dummy. I can edit accordingly
    – cayleaf
    Sep 5, 2023 at 17:21
1

We ran into this as well on Cloudflare, and I'm not sure the other Nginx answers solve the issue because AFAIK, Nginx location blocks do not inherit Cache-Control headers from their parents, so adding a map at the top to set headers did not affect Nginx error_page blocks. This is what worked for us:

location ~* \.(?:css|eot|ico|js|jpg|jpeg|gif|png)$ {
  error_page 404 = @not_found; # If 404, redirect to the named location @not_found.

  # Cache fingerprinted static assets for 30 days.
  # See https://developers.cloudflare.com/cache/concepts/default-cache-behavior.
  add_header Cache-Control 'public,max-age=2592000,must-revalidate,proxy-revalidate';
}

location @not_found {
  add_header Cache-Control 'private,no-cache,no-store' always; # Always set Cache-Control header for 404 responses.
}
-1

my five cents on the issue -

in our PHP project we have few 404 pages, so I decide to do it on PHP level using PHP header() functions

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .