Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server:

tcpdump | grep google

It returns that google can access my server with IP or but there is not anything in access_log or error_log or other apache logs.

cat access_log | grep google or cat error_log | grep

Other bot (bing,...) can access apache but google can't.

There is no problem in my robots.txt or its permissions because as you know robots.txt is not necessary so I delete it but again webmaster tools returned "Unreachable robots.txt", not 404 not found!

Information about server:

  • Server OS : CentOS 6
  • Web Server : Apache 2.x
  • Firewall : IPTables is stoped
  • SELinux is Disabled
  • There is not any think else for security on my server.

How can I investigate the problem and is there any other command that can help me to find the problem ?

share|improve this question

You deleted the robots.txt file, of course it's "unreachable" - the 404 error that Google's getting in response to the request for it is because the file doesn't exist.

If the tool requires a robots.txt to exist, then you probably need one.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.