Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Site are crawled by anonymous bot hosted on amazon ec2. This robot doesn't respect robots.txt and creates high load on web server so I added check if reverse IP for request ends with "amazonaws.com" then server returns 403 page immediately.

This solved problem but may be it can cause other problems? ec2 may be used for some "good" bots and this will cause access problem for theirs. Can you give example of such problems?

share|improve this question
3  
Have you contacted Amazon? Perhaps the behaviour you're seing is a violation of their TOS (and grounds to shut down the misbehaving bot). –  Dana the Sane Sep 15 '11 at 19:27
    
not yet, communication with big companies usually requires lot of time –  valodzka Sep 15 '11 at 19:50
add comment

closed as not constructive by EEAA, squillman, Zoredache, SvW, Shane Madden Sep 15 '11 at 19:28

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

2 Answers

up vote 5 down vote accepted

Amazon EC2 is a hosting platform. They don't directly control what people host. If you block the whole *.amazonaws.com domain then you will stop access to any hosted service using EC2. Which is quite a lot these days.

share|improve this answer
add comment

Check out this similar question: it shows how to block by user agent directly in the .htaccess file. This is good for robots that fail to follow your robots.txt rule...

Blocking by user-agent string in httpd.conf not effective

And you would put that in either the httpd.conf file, OR a .htaccess.

Good luck.

share|improve this answer
1  
If a robot is evil enough to ignore a robot.txt, why do you believe that it wouldn't just provide a forged agent so it appears to be Internet Explorer? –  Zoredache Sep 15 '11 at 19:35
    
Because with .htaccess you block by user agent and/or IP, and doing such, do you think a robot will make more than 5 user agents? If it's actually a robot, it doesn't care who you are. It just goes. It's not going to make a new rule just for your site because it's blocked. AND plus: if it does choose the IE bot, he can just contact Amazon and let them take care of it. I'm sure he's not the ONLY one with this issue. –  U4iK_HaZe Sep 15 '11 at 19:41
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.