Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

What is the best way to prevent someone abusing access to your site using scripts?

share|improve this question

closed as too broad by Jenny D, mdpc, kasperd, Jim B, HBruijn Jan 13 at 23:07

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

    
Your question seems somewhat unclear. Are you talking about people who do have some access you have given them which they are abusing or are you talking about automatc log in / SQL injection / comment spam etc? – Devin Ceartas Sep 25 '09 at 22:19
    
I am talking about people who do have access but they use script and take down the site by accessing it a lot of time within a second – Yogi Sep 25 '09 at 22:29
    
I think you're talking about people running scripts on their own computers accessing yours through the internet. Not local, logged in access. Right? – Dennis Williamson Sep 25 '09 at 22:55

Provide a sensible API, and heavily hint at guidelines for acceptable web scraping. mod_evasive can block people who neglect to add in a sleep routine and flood your servers.

Trying to prevent webscraping itself is a folly -- a computer put your web page data together for human viewing, a computer can pick it apart. Attempting to put that genie back in the bottle will harm the vast majority of existing users, and only provide a speedbump against your target.

share|improve this answer

What you seem to be talking about is a Denial of Service attack. This is merely a question of available resources (available within a specified time frame).

So, as jldugger suggested, you will want to use *mod_evasive* to configure your webserver to only accept a certain number of requests from an originating system. However, this does not stop the bad people from launching a Distributed DoS attack. You could avoid this by increasing your available computing resources and distributing your load over a number of computers.

Your best bet would actually be paranoia. Deny services to everyone except those on a specific white-list. Of course, this only works if this is not a public site.

share|improve this answer

mod_evasive can still let through a lot of undesired traffic. I explain more about it's weaknesses in another answer.

I have seen http://www.fail2ban.org/ recommended, which would address the shortcomings of mod_evasive, but I have not yet tried it myself.

share|improve this answer