3

Site are crawled by anonymous bot hosted on amazon ec2. This robot doesn't respect robots.txt and creates high load on web server so I added check if reverse IP for request ends with "amazonaws.com" then server returns 403 page immediately.

This solved problem but may be it can cause other problems? ec2 may be used for some "good" bots and this will cause access problem for theirs. Can you give example of such problems?

Improve this question
2
  • 3
    Have you contacted Amazon? Perhaps the behaviour you're seing is a violation of their TOS (and grounds to shut down the misbehaving bot).
    – Dana the Sane
    Sep 15, 2011 at 19:27
  • 1
    not yet, communication with big companies usually requires lot of time
    – valodzka
    Sep 15, 2011 at 19:50

2 Answers 2

Reset to default
5

Amazon EC2 is a hosting platform. They don't directly control what people host. If you block the whole *.amazonaws.com domain then you will stop access to any hosted service using EC2. Which is quite a lot these days.

Improve this answer
1

Check out this similar question: it shows how to block by user agent directly in the .htaccess file. This is good for robots that fail to follow your robots.txt rule...

Blocking by user-agent string in httpd.conf not effective

And you would put that in either the httpd.conf file, OR a .htaccess.

Good luck.

Improve this answer
2
  • 2
    If a robot is evil enough to ignore a robot.txt, why do you believe that it wouldn't just provide a forged agent so it appears to be Internet Explorer?
    – Zoredache
    Sep 15, 2011 at 19:35
  • Because with .htaccess you block by user agent and/or IP, and doing such, do you think a robot will make more than 5 user agents? If it's actually a robot, it doesn't care who you are. It just goes. It's not going to make a new rule just for your site because it's blocked. AND plus: if it does choose the IE bot, he can just contact Amazon and let them take care of it. I'm sure he's not the ONLY one with this issue.
    – U4iK_HaZe
    Sep 15, 2011 at 19:41

Not the answer you're looking for? Browse other questions tagged .