Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've blocked a majority of bots that keep sending POST requests to my website, using .htaccess.

Each time one of these bots tries to access my website it receives a 403 forbidden error message.

My question is, why is my bandwidth usage still increasing if I've blocked them in my .htaccess file?

I was always under the impression that web hosting bandwidth is measured by the amount of data that my server sends, not by the amount that it receives.

Is there a way to configure my .htaccess file to just ignore these bots and not send back a status code?

By the way, I'm using a shared server with "unlimited" bandwidth, but the amount of bandwidth that these bots are wasting is rediculous.

Thanks!

share|improve this question
add comment

4 Answers

up vote 6 down vote accepted

The .htaccess files tells your server what specific reply should be sent to these bots. So the request is still happening, and you are still sending back data (the 403 message).

You have no way to prevent the botnet request to reach you, only your provider can block it before it reaches your server. However, you can send nothing back by simply closing the connection for this IP. I'm not sure if Apache has a module to do that, otherwise you can use a software firewall like iptables to do that.

share|improve this answer
add comment

By the time your .htaccess file takes effect, the client's payload has already passed the network interface of your server. From the point of view of your ISP there is basically no difference in in- or outgoing traffic from your server. You or your provider will have to pay it anyway.

If you don't want any data transfer happen, you'll need to block the clients in the border router of your ISP (or at least in a packet filter which is not running on the server itself).

share|improve this answer
add comment

As per the other answers, once the bot gets to the stage of getting a 403 response back, you have both sent and received data.

It'd probably be better to not respond at all, if you can find an apache module to simply drop the connection.

If you have control of the host AND use SYN cookies, it may be worth to also route all IPs you block to 127.0.0.1 (or, maybe even better, add them to a DROP rule in an iptables chain).

share|improve this answer
1  
extending Vatine's answer, you could setup Squid ( or something similar ) and setup acls –  bronzebeard Oct 31 '10 at 12:11
add comment

There are several approaches you can take. One is setting up firewall rules via iptables. Second is disallowing these bots on robot.txt

share|improve this answer
    
robots.txt is purely advisory - any robot that wants to ignore it, can. –  Cyclops Jun 8 '11 at 19:42
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.