I'm getting a lot of requests turning up in our apache logs that look like this
www.example.com:80 10.240.1.8 - - [06/Mar/2013:00:39:19 +0000] "-" 408 0 "-" "-" -
There seems to be no request and no user agent. Has anyone seen this before?
Are you by any chance running your web servers in Amazon behind an Elastic Load Balancer?
Some of the solutions from that forum thread:
And from this blog post:
Something is connecting to the port and then never sending data. HTTP 408 is a "timeout" error. There's a good writeup here: http://www.checkupdown.com/status/E408.html
There are various reasons for a 408 Timeout. But lets start from a premise that everything's fine then at some point these 408's start appearing in your access log - i.e. 408 0 "-" "-".
Like many point out on the net, a 408 represents a connection is made but no request is sent in the appropriate time scale, therefore the server drops the connection with a 408. One arrogant individual actually responded to someone asking for help on this issue with - "What part of Timeout do you not understand".
I think that is very much a novice answer and demonstrates a total lack of understanding of how some security methods function with web-server software.
So back to the beginning, why am I seeing all these 408's. One thing you will have in common with the rest of us managing a server is the vast amount of attacks you receive on a daily basis. Now, what do you do about these? Well: you employ your chosen security methods to deal with them, this is what changes.
Let’s take a very easy example, Drop an IP address. Included in an iptabes file (rules.v4) you have "-A ufw-user-input -s 22.214.171.124 -j DROP". So along comes 126.96.36.199 the firewall recognises the IP and drops the connection. In a lot of configurations you wouldn't even know it's knocked on the door.
Now lets take a more advanced example, Drop the connection based on some criteria. Included in an iptabes file (rules.v4) you have "-A INPUT -p tcp -m tcp --dport 80 -m string --string "cgi" --algo bm --to 1000 -j DROP". This is different because in this iptable rule we are saying look at the first 1000 bytes of the request string and see if you can find a sub-string of "cgi" and if you do find that sub-string then don't go any further, just drop the connection.
Here the security method is good, but as far as your logs are concerned misleading. The 408 0 "-" "-" generated is the best apache can do under these circumstances. The connection was made and the request had to be accepted up to a point in order to apply the string comparison rule which ultimately results in a 408 because your rule met the criteria for the connection to be dropped. So, our little novice darlings couldn't be more wrong if they tried. A connection was made and a request was received (you just won't have visibility of it under these circumstances). Although a 408 is generated it is not a 'Timeout'; your server simply dropped the connection after the request was made in association with your firewall rule. There are many other rules, which would create the same 408 situation. Don't accept too literally the Server Error Code 408 explanation - 'Request Timed Out'.
Ideally there would be another Apache generated Error Code, for example - '499' which would mean 'Server Read Your Request And Decided It Just Couldn't Be Bothered Entertaining You - Sod Off HaHa'.
With the latest web-server software you can practically exclude DOS attacks and the new gene of browsers incorporating predictive capabilities don't cause this problem as some have suggested.
In short, the 408 is generated because the server did not respond to the request, so as far as the client is concerned the connection timed out, when in reality the server read the request but dropped the connection for reasons other than a timeout waiting for a request.
We had this very problem, and puzzled over it for quite a while. The best solution we came up with was suggested by AWS support's ELB team. It essentially hinges on ensuring that your httpd server's timeout settings are all greater than your ELB
We found that the
A colleague recently remarked that while my last post gave a valid explanation how a 408 could have an association with a security measure, it offered no solution.
Piped Access log is my personal solution.
The following should work out-of-the-box on most Ubuntu configurations, and with minimal tinkering on other Apache configurations. I’ve chosen PHP because it’s the easiest to understand. There are two scripts: the first prevents a 408 being written to your access log. The second script sends all 408s to a separate log file. Either way the result is no more 408s in your access log. It's your choice which script to implement.
Use your favorite text editor, I use nano. Open the file where you have your 'LogFormat' and 'CustomLog' directives. Comment-out the originals with the usual # and add the following. You may find these directives in the file below.
sudo nano /etc/apache2/sites-available/default
NOTE: I don't log images in my access log. In my etc/apache2/httpd.conf file I include the line
If this is of no interest to you then remove the
Now create one of the following PHP scripts (
sudo nano /var/log/apache2/PipedAccessLog.php
sudo nano /var/log/apache2/PipedAccessLog.php
Having saved the
Finally to get everything working you need to restart the Apache service. Execute the following at the $ prompt.
If your Apache logs are located elsewhere then change the paths to suit your configuration. Good Luck.
I have found that 408 errors are increasing in both number and frequency. The range of IP addresses they are originating from is also growing (these are logged to their own separate file). There are also evident log patterns displaying consecutive 408's from the same groups of IP that are not due to normal server timeouts in that the originator is attempting connections about 2 or 3 seconds apart in a cyclic pattern (there is no waiting for a timeout before another connection attempt) I see these as simple DDOS style connection attempts. In my opinion these are a type of confirmation message for the originator that a server is online... then they come back later with different tools.... If you increase your timeout interval, you are simply giving them a larger time_allocation to run their hack programs within.