Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm getting a lot of requests turning up in our apache logs that look like this

www.example.com:80 10.240.1.8 - - [06/Mar/2013:00:39:19 +0000] "-" 408 0 "-" "-" -

There seems to be no request and no user agent. Has anyone seen this before?

share|improve this question
    
blog.spiderlabs.com/2011/07/… –  quanta Mar 6 '13 at 5:55
add comment

5 Answers 5

up vote 5 down vote accepted

Are you by any chance running your web servers in Amazon behind an Elastic Load Balancer?

It seems they generate a lot of 408 responses due to their health checks.

Some of the solutions from that forum thread:

  • RequestReadTimeout header=0 body=0

    This disables the 408 responses if a request times out.

  • Change the ELB health check to a different port.
  • Disable logging for the ELB IP addresses with:

    SetEnvIf Remote_Addr "10\.0\.0\.5" exclude_from_log
    CustomLog logs/access_log common env=!exclude_from_log
    

And from this blog post:

  • Adjust your request timeout to be 60 or above.
share|improve this answer
add comment

Something is connecting to the port and then never sending data. HTTP 408 is a "timeout" error. There's a good writeup here: http://www.checkupdown.com/status/E408.html

share|improve this answer
add comment

We had this very problem, and puzzled over it for quite a while. The best solution we came up with was suggested by AWS support's ELB team. It essentially hinges on ensuring that your httpd server's timeout settings are all greater than your ELB idle timeout setting (which defaults to 60 seconds).

  • Ensure that your apache Timeout directive is double than the idle timeout setting of your ELB.
  • Turn on the KeepAlive feature, ensure that MaxKeepAliveRequests is very large (either 0 for infinite or very high, like 2000), and that KeepAliveTimeout is greater than your ELB's idle timeout.

We found that the KeepAlive (and associated settings) setting specifically reduced the quantity of 408s to effectively 0 (we see a few, but very few).

share|improve this answer
    
Unfortunately, I'm using the Elastic Beanstalk. I do not have these options available to me. –  Erick Robertson Apr 13 at 13:39
add comment

There are various reasons for a 408 Timeout. But lets start from a premise that everything's fine then at some point these 408's start appearing in your access log - i.e. 408 0 "-" "-".

Like many point out on the net, a 408 represents a connection is made but no request is sent in the appropriate time scale, therefore the server drops the connection with a 408. One arrogant individual actually responded to someone asking for help on this issue with - "What part of Timeout do you not understand".

I think that is very much a novice answer and demonstrates a total lack of understanding of how some security methods function with web-server software.

So back to the beginning, why am I seeing all these 408's. One thing you will have in common with the rest of us managing a server is the vast amount of attacks you receive on a daily basis. Now, what do you do about these? Well: you employ your chosen security methods to deal with them, this is what changes.

Let’s take a very easy example, Drop an IP address. Included in an iptabes file (rules.v4) you have "-A ufw-user-input -s 37.58.64.228 -j DROP". So along comes 37.58.64.228 the firewall recognises the IP and drops the connection. In a lot of configurations you wouldn't even know it's knocked on the door.

Now lets take a more advanced example, Drop the connection based on some criteria. Included in an iptabes file (rules.v4) you have "-A INPUT -p tcp -m tcp --dport 80 -m string --string "cgi" --algo bm --to 1000 -j DROP". This is different because in this iptable rule we are saying look at the first 1000 bytes of the request string and see if you can find a sub-string of "cgi" and if you do find that sub-string then don't go any further, just drop the connection.

Here the security method is good, but as far as your logs are concerned misleading. The 408 0 "-" "-" generated is the best apache can do under these circumstances. The connection was made and the request had to be accepted up to a point in order to apply the string comparison rule which ultimately results in a 408 because your rule met the criteria for the connection to be dropped. So, our little novice darlings couldn't be more wrong if they tried. A connection was made and a request was received (you just won't have visibility of it under these circumstances). Although a 408 is generated it is not a 'Timeout'; your server simply dropped the connection after the request was made in association with your firewall rule. There are many other rules, which would create the same 408 situation. Don't accept too literally the Server Error Code 408 explanation - 'Request Timed Out'.

Ideally there would be another Apache generated Error Code, for example - '499' which would mean 'Server Read Your Request And Decided It Just Couldn't Be Bothered Entertaining You - Sod Off HaHa'.

With the latest web-server software you can practically exclude DOS attacks and the new gene of browsers incorporating predictive capabilities don't cause this problem as some have suggested.

In short, the 408 is generated because the server did not respond to the request, so as far as the client is concerned the connection timed out, when in reality the server read the request but dropped the connection for reasons other than a timeout waiting for a request.

share|improve this answer
    
But WHY did it drop the connection? We're seeing this is server logs, not client logs. –  Erick Robertson Apr 13 at 13:33
add comment

A colleague recently remarked that while my last post gave a valid explanation how a 408 could have an association with a security measure, it offered no solution.

Piped Access log is my personal solution.

The following should work out-of-the-box on most Ubuntu configurations, and with minimal tinkering on other Apache configurations. I’ve chosen PHP because it’s the easiest to understand. There are two scripts: the first prevents a 408 being written to your access log. The second script sends all 408s to a separate log file. Either way the result is no more 408s in your access log. It's your choice which script to implement.

Use your favorite text editor, I use nano. Open the file where you have your 'LogFormat' and 'CustomLog' directives. Comment-out the originals with the usual # and add the following. You may find these directives in the file below.

sudo nano /etc/apache2/sites-available/default

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" AccessLogPipe

CustomLog "|/var/log/apache2/PipedAccessLog.php" AccessLogPipe env=!dontlog

NOTE: I don't log images in my access log. In my etc/apache2/httpd.conf file I include the line

SetEnvIfNoCase Request_URI ".(gif)|(jpg)|(png)|(css)|(js)|(ico)$" dontlog

If this is of no interest to you then remove the env=!dontlog from the CustomLog directive.

Now create one of the following PHP scripts (#!/usr/bin/php is a reference to the location of the interpreter, make sure that the location is correct for your system - you can do this by typing at the $ prompt; whereis php - this should return something like php: /usr/bin/php /usr/bin/X11/php /usr/share/man/man1/php.1.gz. As you can see #!/usr/bin/php is right for my setup).

sudo nano /var/log/apache2/PipedAccessLog.php

#!/usr/bin/php
<?php
  $file = '/var/log/apache2/access.log';
  $no408 = '"-" 408 0 "-" "-"';
  $stdin = fopen ('php://stdin', 'r');
  ob_implicit_flush (true);
  while ($line = fgets ($stdin)) {
    if($line != "") {
      if(stristr($line,$no408,true) == "") {
        file_put_contents($file, $line, FILE_APPEND | LOCK_EX);
      }
    }
  }
?>

sudo nano /var/log/apache2/PipedAccessLog.php

#!/usr/bin/php
<?php
  $file = '/var/log/apache2/access.log';
  $file408 = '/var/log/apache2/408.log';
  $no408 = '"-" 408 0 "-" "-"';
  $stdin = fopen ('php://stdin', 'r');
  ob_implicit_flush (true);
  while ($line = fgets ($stdin)) {
    if($line != "") {
      if(stristr($line,$no408,true) != "") {
        file_put_contents($file408, $line, FILE_APPEND | LOCK_EX);
      }
      else {
        file_put_contents($file, $line, FILE_APPEND | LOCK_EX);
      }
    }
  }
?>

Having saved the PipedAccessLog.php script; make sure that root has ownership by executing the following at the $ prompt.

sudo chown -R root:adm /var/log/apache2/PipedAccessLog.php

The PipedAccessLog.php script will need read/write and execute permissions so execute the following at the $ prompt.

sudo chmod 755 /var/log/apache2/PipedAccessLog.php

Finally to get everything working you need to restart the Apache service. Execute the following at the $ prompt.

sudo service apache2 restart

If your Apache logs are located elsewhere then change the paths to suit your configuration. Good Luck.

share|improve this answer
    
If the 408's are happening, we need to figure out why and get rid of them. Simply preventing them from being logged or piping them to another file is a waste of server time. It also serves to only hide the problem. It's like cleaning your room by shoving everything under your bed. –  Erick Robertson Apr 13 at 13:41
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.