Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

In my server logs (Apache 2.2.3), I get requests from the useragent "Mozilla/4.0 (compatible;)". Some quick googling seems to suggest that these requests are from proxies that are trying to prefetch pages, but the information is not very conclusive.

Does anyone have experience with these? Do you know where those requests really come from? Is it safe to ignore them if they are 404s?

Sample logs:

1.1.1.1 - - [10/Jun/2010:01:55:01 -0400] "GET /popup/authentication HTTP/1.1" 404 1831 "-" "Mozilla/4.0 (compatible;)"

2.2.2.2 - - [06/Jul/2010:08:50:27 -0400] "GET /images/social/twitter.1.png HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible;)"

3.3.3.3 - - [08/Jul/2010:14:58:41 -0400] "GET /calendar/list/json/6/27/2010 HTTP/1.1" 404 46 "-" "Mozilla/4.0 (compatible;)"

4.4.4.4 - - [11/Jun/2010:13:41:27 -0400] "GET / HTTP/1.1" 200 3875 "-" "Mozilla/4.0 (compatible;)"
share|improve this question

1 Answer 1

up vote 6 down vote accepted

According to several resources (such as this entry), these are requests used by proxy tools that employ pre-fetching strategies (they go and download all of the pages that are linked from the page a user is visiting in order to cache them for fast retrieval). Blue Coat, Verizon, LookSmart, and others have all been found at one time or another to use that UserAgent in their tools.

Also, according to Microsoft, some versions of Internet Explorer, for historical reasons, have used variations of the Mozilla/4.0 (compatible) UserAgent. See here for more details.

share|improve this answer
3  
Actually, all versions of Internet Exploder up to IE8 identify as Mozilla/4.0 (compatible), but there's usually more within the brackets, like Trident/OS version numbers. IE9 identifies as 5.0 compatible, like all other modern browsers –  fahadsadah Jul 9 '10 at 15:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.