I rent a server somewhere out in Canada or so that I'm using to host a website of mine. That website has close to 400,000 pages that I wanted to index today. For that, I wrote a crawler a while back (see JCrawler on Stackoverflow.com). Now, I'm greedy and didn't want it to take too long so I ran multiple threads resulting in some 60+ requests per second from my IP. A couple minutes later, my server locked me out. I can still FTP into it, but I can't HTTP it.

As server administrator or user, do you have any idea how servers usually handle these situations? Is it common to place a permanent or temporary ban on the IP or what is typically done?

Naturally, I'll re-run my software with fewer requests once I'm back on.

  • There are many ways to do this but we cannot possibly know what system is used on your particular web server. – John Gardeniers Jun 6 '10 at 1:45

Take a look at mod_evasive. It is probably what locked you out based on your description.


What did lock you out?

It is your server, so you should know / check with your provider before coming here. Can be some auto firewall on the server that you are not aware of (nitpick: says a lot about your ability as administrator) or some provider firewall thingy that locks you out (in which case it is actually not the server doing anything).

  • Judging by your comment, I guess I'm glad I'm not the administrator. I just pay for hosting. I finally gave in and called them (PITA because I'm currently in Europe; I wanted to avoid an international call), so let's make this a general kind of question. I was unable to access the server via HTTP request. FTP worked fine and any other IP was able to access the server. – Jan Kuboschek Jun 5 '10 at 18:23
  • Then syou should not have said "I rent a server" which implies a physical box that you administrate ;) – TomTom Jun 6 '10 at 3:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.