Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

The problem:

I manage a website which has lot's of dynamically generated pages. Every day bots from Google, Yahoo and other search engines download like 100K+ of pages. And sometimes i have problems with "hackers" trying to massively download the whole website.

I would like to block ip addresses of "hackers" while keeping search engine bots crawling the pages. What is the best way to do this ?

Right now i am solving the problem as follows. I save ip of each page request to a file every X seconds. And i have a crontab script which counts repetitive ip's every 30 minutes. For ip's which are repeated too many times, the script checks a hostname- if it doesn't belong to Google/Yahoo/Bing/etc then we have a candidate for banning. But i don't really like my solution and think that auto banning could be done better or using some out of the box solution.

share|improve this question

You didn't state your OS, so I will happily tell you the OpenBSD version: in pf.conf place something like the following in your ruleset (for 100 conns per 10 secs max):

table <bad_hosts> persist
block quick from <bad_hosts>
pass in on $ext_if proto tcp to $webserver port www keep state \
                 (max-src-conn-rate 100/10, overload <bad_hosts> flush global)

you could add a whitelist and a cron job kicking addresses from bad_hosts after a day or two.

share|improve this answer

I would have thought fail2ban is the answer.

You can use white lists to stop search engines getting blocked.

share|improve this answer

Have a look at Simple Event Correlator. It can automatically run commands (ie. add a block to iptables) after a certain amount of lines matching a regular expression have been seen within a window of time. It can also define a "context" that expires. When a context expires you can unblock the IP in question (ie. remove from iptables).

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.