Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm going to block all bots except the big search engines.

One of my blocking methods will be to check for "language": Accept-Language

If it has no Accept-Language the bot's IP adress will be blocked untill 2037.

Googlebot do not have Accept-Language, I want to verify it with DNS lookup

<?php
    gethostbyaddr($_SERVER['REMOTE_ADDR']);
?>

Is it ok to use gethostbyaddr, can someone pass my "gethostbyaddr protection"?

share|improve this question

1 Answer 1

up vote 2 down vote accepted

This is really a question for Stack Overflow more than Server Fault (a SEO site would actually be best), but what you've got there is very unreliable. The googlebot could come from any numbers of IP addresses and thus reverse DNS lookups. There could be thousands of different options, and if one changes, then bam you've blocked a legitimate bot for almost 30 years.

In fact, I just found a Stack Overflow question here that answers your question. It was the first google hit for "How to identify Google Bot".

You identify search engines by user agent and IP address. More info can be found in How to identify search engine spiders and webbots. It's also worth noting this list. You shouldn't treat user agents (or even remote hosts) as necessarily definitive however. User agents are really nothing more than what the other end tells you it is and it is of course free to tell you anything. It's trivial to write code to pretend to be Googlebot.

(Note: Google actually suggest that you use RDNS to verify their bots. This may be OK for their scenario, but it is certainly not a good idea for wider use. Plus it's very slow).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.