Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server:

tcpdump | grep google

It returns that google can access my server with IP aa.bb.cc.xx or aa.bb.cc.yy. but there is not anything in access_log or error_log or other apache logs.

cat access_log | grep google or cat error_log | grep aa.bb.cc.xx

Other bot (bing,...) can access apache but google can't.

There is no problem in my robots.txt or its permissions because as you know robots.txt is not necessary so I delete it but again webmaster tools returned "Unreachable robots.txt", not 404 not found!

Information about server:

  • Server OS : CentOS 6
  • Web Server : Apache 2.x
  • Firewall : IPTables is stoped
  • SELinux is Disabled
  • There is not any think else for security on my server.

How can I investigate the problem and is there any other command that can help me to find the problem ?

share|improve this question
add comment

1 Answer

You deleted the robots.txt file, of course it's "unreachable" - the 404 error that Google's getting in response to the request for it is because the file doesn't exist.

If the tool requires a robots.txt to exist, then you probably need one.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.