How to Avoid Being Scraped

Question

We have a searchable DB, we limit the results to 15 per page and only 100 results and yet still get people trying to scrape the site. We ban sites that hit it fast enough but i was wondering if there is anything else that we can do. Flash render the results maybe?

Answer 1

You could make it a bit more difficult by retrieving the records via AJAX, and using an authentication ID (like an API key) for the AJAX calls.

Of course you can get around this by reading the ID and then making the AJAX request using that.

Rndering with Flash is an alternative as you point out (not 100% un-scrappable do), as is rendering in PDF.

Answer 2

Since there is obviously a demand for your database, have you thought about turning it around and providing what the scrapers want? Form a business connection with the scrapers and encourage appropriate use with an API?

Answer 3

There is some good info in How do you stop scripters from slamming your website hundreds of times a second?

Answer 4

There is no technological solution to prevent a motivated individual from scraping your publicly accessible content.

You can, however, legally protect your intellectual property by:

• Ensuring that your site has a clearly marked copyright
• Posting a Terms of Service in the footer that clearly prohibits scraping
• Consider embedding a digital watermark into all of your site's content. Don't forget that text can be watermarked as well!

Answer 5

How about setting up authentication (and perhaps captcha), tracking usage, and limiting access to some number of records or searches is a given time period?

Answer 6

You will probably find that the scrapers will improve their came as you apply different techniques. Perhaps there is a way to analyze the behavior of users who scrape and present a captcha or other disruption? Perhaps you could limit the results to a smaller number for a period of time to force the scrapers to wait for 10 days. If they don't log on in between then assume they are scapers?

Whatever you do, make sure to mix up your techniques to give them a little more longevity.

Answer 7

You need to consider that the scrapers may not be using your web pages and forms, they may just be calling your site at a http level.

I think that the best solution would be to throw up a catchpa after an ip address requests more than a certain request threshold.

You need to be VERY careful though to ensure that you do not affect the scalability of you applicaiton for real users.

Limiting the amount of data per page as you describe in the question will only increase the number of requests that the clients will make against your server.

Answer 8

Make sure you have a robots.txt ... yeah I know not everyone honors it .. but some still do