When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server:

tcpdump | grep google

It returns that google can access my server with IP aa.bb.cc.xx or aa.bb.cc.yy. but there is not anything in access_log or error_log or other apache logs.

cat access_log | grep google or cat error_log | grep aa.bb.cc.xx

Other bot (bing,...) can access apache but google can't.

There is no problem in my robots.txt or its permissions because as you know robots.txt is not necessary so I delete it but again webmaster tools returned "Unreachable robots.txt", not 404 not found!

Information about server:

  • Server OS : CentOS 6
  • Web Server : Apache 2.x
  • Firewall : IPTables is stoped
  • SELinux is Disabled
  • There is not any think else for security on my server.

How can I investigate the problem and is there any other command that can help me to find the problem ?


You deleted the robots.txt file, of course it's "unreachable" - the 404 error that Google's getting in response to the request for it is because the file doesn't exist.

If the tool requires a robots.txt to exist, then you probably need one.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.