Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I've configured a server running apache and a couple of days ago I noticed in the logs that there are bots running endless queries to the site. The logs show that those bots are running about 60 queries per second for about 20 minutes coming from the same IP address. How can I limit the queries and what's the appropriate way of dealing with such bots?

Thanks in advance.

share|improve this question

4 Answers 4

up vote 4 down vote accepted

Consider installing a rate limiting software, it will help you to defend not only against lawful bots. You can use mod_evasive for Apache, or you can install Nginx as a frontend and use its HttpLimitZoneModule, it is built in.

share|improve this answer

I highly recommend the tool fail2ban (http://www.fail2ban.org/); we use it on our own servers and it is amazingly useful for limiting DoS floods for more than just Apache.

fail2ban actively monitors various logs for different patterns (we have ours set to just monitor Apache hits), and if it determines that there are too many hits in a certain amount of time from a specific IP, it will ban that IP for a period you determine (we use a 20 minute ban window)

fail2ban has pre-built packages for most distribution, or you can grab the source; being Python, you don't even need to compile it! :D

share|improve this answer
    
This seems really interesting! As from what I understood this will add banned IPs to IPTABLES which is great for me. Can you tell me how much resources does this tool need to run? Does it use high cpu/mem if the log file is long ? (100-200mb) –  tftd Feb 3 '11 at 23:24
1  
We've never noticed it impacting our server resources, even under extremely heavy traffic; of course, your mileage may vary :P –  jonodlo Feb 4 '11 at 0:09
    
Thank you very much for the help :) –  tftd Feb 4 '11 at 14:12

A simple solution would be to edit your robots.txt file and add the following line:

User-agent: *
Crawl-delay: 60

Changing the crawl-delay to a number in seconds between requests. "Good" bots should listen to this, and wait between requests. The "bad" robots don't listen to any robots.txt rules however..

share|improve this answer
    
This works only with nice bots. A bot such as the one mentioned in the question is probably not well-behaved. –  bortzmeyer Mar 6 '12 at 9:23

robots.txt and redirecting the bots to static sites. Generally it's a good thing to have bots (search engines) crawling (indexing) your site. The suggested methods help you control how they index it though.

share|improve this answer
    
Yes it is but we're not talking about search/indexing bots. This bot is trying to open only one page all the time. :) –  tftd Feb 3 '11 at 18:02
    
You're initial description made it sound like multiple bots to multiple pages. I'd recommend then blocking the IP using a firewall. –  frogstarr78 May 12 '11 at 15:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.