1

I've configured a server running apache and a couple of days ago I noticed in the logs that there are bots running endless queries to the site. The logs show that those bots are running about 60 queries per second for about 20 minutes coming from the same IP address. How can I limit the queries and what's the appropriate way of dealing with such bots?

Thanks in advance.

4 Answers 4

4

Consider installing a rate limiting software, it will help you to defend not only against lawful bots. You can use mod_evasive for Apache, or you can install Nginx as a frontend and use its HttpLimitZoneModule, it is built in.

3

I highly recommend the tool fail2ban (http://www.fail2ban.org/); we use it on our own servers and it is amazingly useful for limiting DoS floods for more than just Apache.

fail2ban actively monitors various logs for different patterns (we have ours set to just monitor Apache hits), and if it determines that there are too many hits in a certain amount of time from a specific IP, it will ban that IP for a period you determine (we use a 20 minute ban window)

fail2ban has pre-built packages for most distribution, or you can grab the source; being Python, you don't even need to compile it! :D

2
  • This seems really interesting! As from what I understood this will add banned IPs to IPTABLES which is great for me. Can you tell me how much resources does this tool need to run? Does it use high cpu/mem if the log file is long ? (100-200mb)
    – tftd
    Feb 3, 2011 at 23:24
  • 1
    We've never noticed it impacting our server resources, even under extremely heavy traffic; of course, your mileage may vary :P
    – jonodlo
    Feb 4, 2011 at 0:09
0

A simple solution would be to edit your robots.txt file and add the following line:

User-agent: *
Crawl-delay: 60

Changing the crawl-delay to a number in seconds between requests. "Good" bots should listen to this, and wait between requests. The "bad" robots don't listen to any robots.txt rules however..

1
  • This works only with nice bots. A bot such as the one mentioned in the question is probably not well-behaved.
    – bortzmeyer
    Mar 6, 2012 at 9:23
-1

robots.txt and redirecting the bots to static sites. Generally it's a good thing to have bots (search engines) crawling (indexing) your site. The suggested methods help you control how they index it though.

2
  • Yes it is but we're not talking about search/indexing bots. This bot is trying to open only one page all the time. :)
    – tftd
    Feb 3, 2011 at 18:02
  • You're initial description made it sound like multiple bots to multiple pages. I'd recommend then blocking the IP using a firewall. May 12, 2011 at 15:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .