0
votes
1answer
97 views

prevent search engines indexing depending on domain

We have a dedicated server with a hosting company with a couple of dozens of webs in it. It happens that the nameservers (EG: ns1.domain.com, ns2.domain.com) ip's are coincident with some client webs, ...
3
votes
3answers
828 views

Search Engine for Network Shares

We have a clustered storage server that contains many project files like Corel, Indesign, Edius, Adobe Premiere, etc.. Also it has lots of image, audio and video files. We are using sharepoint and ...
0
votes
1answer
184 views

Can I use a Google Appliance/Mini to crawl and index sites I don't own?

Maybe this is a stupid question, but... I am working with this company and they said they needed to get "permission" to crawl other people's sites. They have a Google Search Appliance And some ...
1
vote
6answers
2k views

What happens if a website does not have a robots.txt file?

If the robots.txt file is missing in the root directory of a website, how are things treated as: the site is not indexed at all the site is indexed without any restrictions It should logically be ...
1
vote
3answers
541 views

odd query strings in Googlebot requests

Google's indexing bot (edit: yes, it's Google, IP resolves) seems to be adding arbitrary query strings to our home page. xx.xxx.xx.xxx - - [30/Jun/2009:10:14:37 -0400] "GET /?key=61680 HTTP/1.1" 200 ...