Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am in charge of production servers serving static content for a website. Those servers are constantly being crawled by bots looking for potential exploits (which isn't that much of a problem security-wise because no application can be reached behind the web server) but generates thousands of 404 per day, sometimes per hour. I am looking into ways of blocking those requests but it's tricky (you want to make sure you don't block legitimate traffic and these bots are becoming more and more clever at looking like they're legit) and is going to take me a while to find an acceptable solution.

In the meantime I would like to reduce the performance impact of serving those 404 pages. Indeed we're using nginx which by default is configured to serve it's 404 page from the disk (This can be changed using the error_page directive but in the end the 404 will either have to be served from disk or from another external source (e.g. upstream application which would be worst)) which isn't ideal.

I ran a test with ab on my local machine with a basic configuration: in one case I echo a message directly from nginx so the disk isn't touched at all, in the other case I hit a missing page and nginx serves its 404 from disk.

server {
  # [...] the default nginx stuff
  location / {

  }
  location /this_page_exists {
     echo "this page was found";
  }
}

Here are the test results (my laptop has Intel(R) Core(TM) i7-2670QM + SSD in case you're wondering why they are so high):

$ ab -n 500000 -c 1000 http://localhost/this_page_exists
Requests per second:    25609.16 [#/sec] (mean)

$ ab -n 500000 -c 1000 http://localhost/this_page_doesnt_exists
Requests per second:    22905.72 [#/sec] (mean)

As you can see, returning a value with echo is 11% ((25609−22905)÷22905×100) faster than serving the 404 page from disk. Accordingly I would like to echo a simple 404 Page not Found string from nginx.

I tried many things so far but they all failed, essentially the idea was this:

location / {
  try_files $uri @not_found;
}
location @not_found {
  echo "404 - Page not found";
}

The problem is that as soon as the echo directive is used, the http response code is set to 200. I tried changing that by doing error_page 200 = 400 but that breaks the configuration.

How can I serve a 404 page directly from nginx? (without hacking the source which may be might next step)

share|improve this question
    
Have you considered using fail2ban to catch and then block these at the firewall ? –  Iain Jun 23 '12 at 11:35
    
It's one of the options I'm looking into but I'm really clueless about it, therefore installing/testing properly + releasing into production is going to take a while. As I said before echoing a 404 would be a In the meantime solution (which could turn out to be worthy of the effort in my opinion). –  user64204 Jun 23 '12 at 11:40
    
@MDMarra: my question is about serving 404 from nginx directly instead of the disk so as to improve performance, and you removed the 404, disk, and performance tags I had set, why? –  user64204 Jun 23 '12 at 12:46
1  
Because they're bad tags and no one should use them. All three are ambiguous and add nothing to the categorization of the question. –  MDMarra Jun 23 '12 at 12:49
add comment

2 Answers 2

up vote 3 down vote accepted

Actually, by default, nginx generates a 404 response internally. It only serves a file from disk if you tell it to using an error_page directive. If you want to control the format of the 404 page, instead of echo "404 - page not found";, you can use return 404 "404 - page not found"; (assuming you're using a somewhat recent version of nginx, I belive you need 0.9 or newer)

share|improve this answer
    
I've tested and it works. I was able to achieve 25329 req/sec with that 404 method so there is definitely a performance improvement compared with the default 404. –  user64204 Jun 23 '12 at 13:17
add comment

Nginx echo module is what you need. But you should use it with error_page:

error_page 404 @echo_404;
location @echo_404 { echo "Not found"; }

You can make nginx close active connection by returning 444:

return 444;

This will close socket immediately without writing anything to the net.

The difference in your calculations is just a result of open_file_cache been not enabled. If you want it go faster tweak your system: accept filters, socket queue and buffers and so.

share|improve this answer
    
About the echo: I already tried it but as soon as I use echo the response code changes to 200. About the 444: that's even better performance wise (28216 req/sec), but I think returning a 404 is still important in that case as some genuine clients might try to access non-existing resources (broken links) in which case a 404 might be nicer than terminating the connection without returning any data –  user64204 Jun 23 '12 at 13:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.