Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there any way to figure out how many bots (even if it's just one type of bot) that hits our site, on a daily, weekly and monthly average?

we have the standard IIS6 web site logs in lots of files.

Any tricks or recommendations?


Program needs to be .NET based. No php or extra ugly packages, please. Free would be a bonus.


locked by HopelessN00b Apr 3 '15 at 12:55

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as off-topic by HopelessN00b Apr 3 '15 at 12:55

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

up vote 0 down vote accepted

Many reporting packages can report on this.

For instance, Awstats says

AWStats is able to detect robots visits.[...] "Robots visitors" are reported separately in the "Robots/Spiders visitors" chart.

EDIT: AWStats is free, written in Perl, and works just fine on Windows and with IIS logs.

I've updated my original post. Bascially, i can't use PHP. has to be .NET based ... and preferably free. – Pure.Krome May 21 '09 at 7:34
AWStats is based on Perl, not PHP. – WerkkreW May 21 '09 at 7:52
AWstats has only very basic bot-detection. Anything that has a non-robot useragent and does not request robots.txt is not recognised as a robot. – Jacco May 21 '09 at 10:55
re: Php/Perl. opps. I ment Perl, but same thing.. i don't want to install extra stuff besides the app iteself. – Pure.Krome May 22 '09 at 5:56

Here is a pretty good list of known bot user-agents. I am not familiar with any free web log analytic tools written in .NET, but you could easily parse the logs yourself if all you are looking for are bots.


Google Analytics anyone? Free, Javascript based, so no need for server side languages. In depth coverage of hits, bots too.

I'm already using Google Analytics. It has stats on it's own robot hits? – Pure.Krome Jun 11 '09 at 0:54
Hm, sorry I misled you, I thought GA did bots too. The bots/crawlers can't execute Javascript, Awstats would work for you since it reads the server logs. – snacsnoc Jun 11 '09 at 2:37
I didn't know this, but Google Analytics actually records how many times it hits your site. That said, it doesn't count the OTHER bots, though. – Pure.Krome Jun 29 '09 at 1:18

I use WebLog Expert when I want to examine our logs. There is a free version (lite) but that doesn't include the scheduler or command line functionality. The software itself runs as a desktop app (either remotely with ability to FTP logs or can be installed on the server), and can output reports in CSV, PDF and HTML.


why not outsourcing your web stats ? With solution like sitemeter, you just put an extra javascript, and they will give you stats on your site.

Web stats may be a pain for admin, doesn't always worth it to implement in house solution


Not the answer you're looking for? Browse other questions tagged or ask your own question.