Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My Google-fu is failing me right now.

I'm trying to figure out whether Google's web crawler downloads non-image binary files when it spiders sites. I know it downloads (and indexes) images and PDFs, but what about .zip, .dmg, etc?

My client offers a lot of sofware packages for download on their site, and they're trying to figure out whether search engines are making up much of the bandwidth involving these files.

share|improve this question
2  
Why not just block the files directory in robots.txt and be sure? Even if they don't currently, nothing's stopping them from adding such a feature in the future. –  ceejayoz Apr 27 '12 at 21:02
    
Definitely a good idea for the future, but the issue I'm dealing with now is that my client has sent me a list of hits on their downloads, and they want to know whether it's people or web crawlers. I'm trying to figure out how to answer this question regarding their existing/past stats. –  jessica Apr 27 '12 at 21:05
    
@ceejayoz Similarly, nothing's stopping them from deciding to ignore binary files in robots.txt some day as it's not an access control mechanism, it's just a suggestion that Google voluntarily opts in to. In a similar vein, Google respects robots.txt but other search engines do not necessarily. –  msanford Apr 27 '12 at 21:30
add comment

2 Answers 2

up vote 6 down vote accepted

The answer to your first question seems to be "maybe":

What file types can Google index?

Google can index the content of most types of pages and files. See the most common file types.

But the link to common files types are all text.

Even if you search for binary files like Windows Installers (.msi), you may get a link to a page containing the file or a direct link to the file, but Google almost certainly decides how to index it based on what is around the link on the page, rather than by downloading and deciphering the binary files' contents.

As to your main question, Google's recommended method way for checking whether the bot hit your site or not is to use a reverse-DNS lookup:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

Keep in mind that Google's mission "is to organize the world’s information and make it universally accessible and useful." This means that they are constantly innovating, attempting to index non-text data in ways that makes it searchable. To expand on ceejayoz's idea that just because they didn't do it yesterday doesn't mean they won't do it tomorrow: Google will do everything they can to be able to it tomorrow!

share|improve this answer
1  
I wonder if Google may do a HEAD request for binary files, as well. –  ceejayoz Apr 27 '12 at 21:31
1  
Good point, but remember to also forward resolve the host name. Anyone who 'owns' an IP address can set a PTR record to a .googlebot.com domain. –  vvanscherpenseel Apr 27 '12 at 21:31
    
@vvanscherpenseel Forgot that one! And here I go just doing whatever Google tells me ;) –  msanford Apr 27 '12 at 21:33
add comment

Instead of taking a guess, why not check the access_logs to see what the User Agent or the requesting host is? That way you can even tell how much bandwidth Google (or other crawlers) are taking, by adding the data traffic per request.

share|improve this answer
1  
As I mention in my answer, user-agents are trivial to spoof. Google's recommended method is to perform a reverse DNS lookup. –  msanford Apr 27 '12 at 21:28
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.