Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I rent a server somewhere out in Canada or so that I'm using to host a website of mine. That website has close to 400,000 pages that I wanted to index today. For that, I wrote a crawler a while back (see JCrawler on Stackoverflow.com). Now, I'm greedy and didn't want it to take too long so I ran multiple threads resulting in some 60+ requests per second from my IP. A couple minutes later, my server locked me out. I can still FTP into it, but I can't HTTP it.

As server administrator or user, do you have any idea how servers usually handle these situations? Is it common to place a permanent or temporary ban on the IP or what is typically done?

Naturally, I'll re-run my software with fewer requests once I'm back on.

share|improve this question
    
There are many ways to do this but we cannot possibly know what system is used on your particular web server. –  John Gardeniers Jun 6 '10 at 1:45
add comment

2 Answers

up vote 1 down vote accepted

Take a look at mod_evasive. It is probably what locked you out based on your description.

share|improve this answer
add comment

What did lock you out?

It is your server, so you should know / check with your provider before coming here. Can be some auto firewall on the server that you are not aware of (nitpick: says a lot about your ability as administrator) or some provider firewall thingy that locks you out (in which case it is actually not the server doing anything).

share|improve this answer
    
Judging by your comment, I guess I'm glad I'm not the administrator. I just pay for hosting. I finally gave in and called them (PITA because I'm currently in Europe; I wanted to avoid an international call), so let's make this a general kind of question. I was unable to access the server via HTTP request. FTP worked fine and any other IP was able to access the server. –  Jan Kuboschek Jun 5 '10 at 18:23
    
Then syou should not have said "I rent a server" which implies a physical box that you administrate ;) –  TomTom Jun 6 '10 at 3:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.