Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We are noticing that a significant amount of web traffic is from content scrapers (determined due to their crawling pattern). They are useless visitors to us but consume a lot of our resources (bandwidth, cpu). Is there any application/firewall to detect content scrapers and block them?

Excluding Search engine crawlers, they are not useless.

Note: I prefer to use existing solutions. It believe this is a common problem and there should be an existing solution.

share|improve this question

closed as not constructive by Brent Pabst, Greg Askew, Sven, Michael Hampton, rnxrx Dec 31 '12 at 23:39

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

up vote 0 down vote accepted

The best way to do this is to block the traffic using netfilter/iptables as this is essentially more performant than blocking via apache2 / php. Problem here is that you're required to know the ip / hostname of the content scrapers.

A possible extension could be you try to detect content scrapers based on their behaiviour (-> statistical methods! - e.g. requests per minute) or e.g. search for missing useragent or other stuff a normal user browser would have and then deny access for them. Of course you could also add the IP / Hostname via php (or whatever environment you use) to iptables so it's blocked. But normally this requires root permission and it's NOT a good idea to give root permission to your apache2.

share|improve this answer
Content scrappers normally provide a legitimate user-agent. Request per/min can not be a good measure to block the IP as some offices have several users behind a proxy/ISA. – newbie Dec 31 '12 at 15:00
I use a system as described above for a customer of mine - and it works fine since half a year :) – Simon Strasser Dec 31 '12 at 15:08
Another thing: It's not just about request/minute but also about continuity of requests and also about mass requests at unusual times. – Simon Strasser Dec 31 '12 at 15:10

Not the answer you're looking for? Browse other questions tagged or ask your own question.