Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a searchable DB, we limit the results to 15 per page and only 100 results and yet still get people trying to scrape the site. We ban sites that hit it fast enough but i was wondering if there is anything else that we can do. Flash render the results maybe?

share|improve this question
    
Make sure you have a robots.txt ... yeah I know not everyone honors it .. but some still do –  trent May 12 '09 at 2:21
add comment

7 Answers

up vote 4 down vote accepted

You could make it a bit more difficult by retrieving the records via AJAX, and using an authentication ID (like an API key) for the AJAX calls.

Of course you can get around this by reading the ID and then making the AJAX request using that.

Rndering with Flash is an alternative as you point out (not 100% un-scrappable do), as is rendering in PDF.

share|improve this answer
add comment

Since there is obviously a demand for your database, have you thought about turning it around and providing what the scrapers want? Form a business connection with the scrapers and encourage appropriate use with an API?

share|improve this answer
1  
Yes I love APIs –  Isaac Waller May 12 '09 at 4:08
    
+1 You can even charge a small fee for API usage and make a few buchs from the scrapers-turned-clients –  Sander Marechal May 12 '09 at 9:47
    
Yes, we setup a timer and ban if the pages were hit too fast, as far as selling the info it seems unlikely, in the past they used the info to put it on another site and go for click revenue. –  Randin May 14 '09 at 2:39
add comment

There is no technological solution to prevent a motivated individual from scraping your publicly accessible content.

You can, however, legally protect your intellectual property by:

  • Ensuring that your site has a clearly marked copyright
  • Posting a Terms of Service in the footer that clearly prohibits scraping
  • Consider embedding a digital watermark into all of your site's content. Don't forget that text can be watermarked as well!
share|improve this answer
add comment

How about setting up authentication (and perhaps captcha), tracking usage, and limiting access to some number of records or searches is a given time period?

share|improve this answer
add comment

You will probably find that the scrapers will improve their came as you apply different techniques. Perhaps there is a way to analyze the behavior of users who scrape and present a captcha or other disruption? Perhaps you could limit the results to a smaller number for a period of time to force the scrapers to wait for 10 days. If they don't log on in between then assume they are scapers?

Whatever you do, make sure to mix up your techniques to give them a little more longevity.

share|improve this answer
add comment

You need to consider that the scrapers may not be using your web pages and forms, they may just be calling your site at a http level.

I think that the best solution would be to throw up a catchpa after an ip address requests more than a certain request threshold.

You need to be VERY careful though to ensure that you do not affect the scalability of you applicaiton for real users.

Limiting the amount of data per page as you describe in the question will only increase the number of requests that the clients will make against your server.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.