Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've been following a few spiders in our logs and I did a traceroute on their ip to find out they are in fact EC2 instances. The user agents are listed as Google bot and msnbot but they are not Google or MS ip's. Is there anything I can do, is spoofing user agents a common practice? I'm guessing that if I ban their ip(which I've done) they will just start a new instance and carry on. I don't want to ban all EC2 instances though.

share|improve this question
up vote 5 down vote accepted

When you really start to delve into logs, you'll find that a huge number of robots do header spoofing; most of them spoofing as IE (some of them unsuccessfully; typos get your agent string spotted fast!).

There's an interesting EFF experiment that looks at uniquely identifying users with the data presented by the browser: Panopticlick. Gathering more info at the application level to attempt to block could potentially get you somewhere, as the non-browser nodes will fail to return some of those fields.

But, in the same sense that blocking their IPs is unlikely to work for long, attempting to block based on user-agent (or any other unique criteria) is unlikely to work for long if they're determined to spider you. In the end, it's not going to be worth your time or energy to try to block every rogue bot on the net; just set up your robots.txt file, keep an eye out for the nasty ones trying to hit you with SQL injection or the like, and rest easy.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.