When I try to fetch my site as a googlebot in webmaster tools it return Unreachable robots.txt, after investigate I understood google bot can see my server:
tcpdump | grep google
It returns that google can access my server with IP aa.bb.cc.xx or aa.bb.cc.yy. but there is not anything in access_log or error_log or other apache logs.
cat access_log | grep google or cat error_log | grep aa.bb.cc.xx
Other bot (bing,...) can access apache but google can't.
There is no problem in my
robots.txt or its permissions because as you know
robots.txt is not necessary so I delete it but again webmaster tools returned "Unreachable robots.txt", not 404 not found!
Information about server:
- Server OS : CentOS 6
- Web Server : Apache 2.x
- Firewall : IPTables is stoped
- SELinux is Disabled
- There is not any think else for security on my server.
How can I investigate the problem and is there any other command that can help me to find the problem ?