Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'll begin by telling you what we do.

The measures we have implemented catch a lot of spiders, but we have no idea how many we are missing. Currently, we apply a set of measures that are obviously partially overlapping:

  1. monitor requests for our robots.txt file: then of course filter all other requests from same IP address + user agent

  2. compare user agent and IP addresses against published lists: and publish the two lists that seem to be the most widely used for this purpose

  3. pattern analysis: we certainly don't have pre-set thresholds for these metrics but still find them useful. We look at (i) page views as a function of time (i.e., clicking a lot of links with 200 msec on each page is probative); (ii) the path by which the 'user' traverses out Site, is it systematic and complete or nearly so (like following a back-tracking algorithm); and (iii) precisely-timed visits (e.g., 3 am each day).

Again, I am fairly sure we're getting the low-hanging fruit, but I'm interested in getting the views from the community.

share|improve this question
up vote -2 down vote accepted

These Newsletter posts tagged as Web Log Analysis at
the commercial Web log Analyzer from Nihuo site pages could be useful reading.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.