As many who posted here, my apache server is getting brutally hammerred to near death by robots, most of which are good robots.

There is no way to change their crawl rate.

Anyone have a suggestion as to how to solve this problem? I thought perhaps of creating 2 groups, called users and bot and allocating 50% for each.

Then from the behaviour or user-agent, if something is identified as a bot, it will be set to the bots group whose members have a maximum of 50% total resources. If bots try to take more, the system will slow down proportiannly so they never have more than 50%.

Does anyone know how to go about doing this or some other method with the same goal? I am using centos and have very little experience in this.

  • 1
    You mentioned yourself that bots are good, so doing what you imply is only going to bottleneck the performance of the bot and potentially result in it not completing a request. I would personally work on improving the performance of your system as a whole rather than trying to do that. – Peter Dec 3 '14 at 6:40
  • @Peter we tried upgrading the hardware but the bots just come in stronger. we would like the server to slow down only for the bots group if they exceed their 50% resources allocated – Ray S. Dec 3 '14 at 6:42
  • That doesn't sound like something legit bots do. Just block their IPs. If you don't want to do that, I stand by my previous comment. – Peter Dec 3 '14 at 6:45
  • @Peter ok, but like i said, they take as much as they can get, so improve the system will not help. we have a site with many tens of thousands of pages with pictures – Ray S. Dec 3 '14 at 6:54
  • Actual numbers might be of help here. If you have a $2/month VPS and robots are chewing up 50% of your CPU, that's entirely different from having robots chew through half the CPU on a dedicated server with 24 cores. Likewise, a metric on the bots like connections/requests per second would go a long way to helping determine if this is normal bot traffic your machine should be able to handle, or if there's something amiss. – HopelessN00b Dec 3 '14 at 6:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.