Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Is there any way to figure out how many bots (even if it's just one type of bot) that hits our site, on a daily, weekly and monthly average?

we have the standard IIS6 web site logs in lots of files.

Any tricks or recommendations?

Update

Program needs to be .NET based. No php or extra ugly packages, please. Free would be a bonus.

share|improve this question

5 Answers 5

up vote 0 down vote accepted

Many reporting packages can report on this.

For instance, Awstats says

AWStats is able to detect robots visits.[...] "Robots visitors" are reported separately in the "Robots/Spiders visitors" chart.

EDIT: AWStats is free, written in Perl, and works just fine on Windows and with IIS logs.

share|improve this answer
    
I've updated my original post. Bascially, i can't use PHP. has to be .NET based ... and preferably free. –  Pure.Krome May 21 '09 at 7:34
    
AWStats is based on Perl, not PHP. –  WerkkreW May 21 '09 at 7:52
    
AWstats has only very basic bot-detection. Anything that has a non-robot useragent and does not request robots.txt is not recognised as a robot. –  Jacco May 21 '09 at 10:55
    
re: Php/Perl. opps. I ment Perl, but same thing.. i don't want to install extra stuff besides the app iteself. –  Pure.Krome May 22 '09 at 5:56

Here is a pretty good list of known bot user-agents. I am not familiar with any free web log analytic tools written in .NET, but you could easily parse the logs yourself if all you are looking for are bots.

http://www.user-agents.org

share|improve this answer

Google Analytics anyone? Free, Javascript based, so no need for server side languages. In depth coverage of hits, bots too.

share|improve this answer
    
I'm already using Google Analytics. It has stats on it's own robot hits? –  Pure.Krome Jun 11 '09 at 0:54
    
Hm, sorry I misled you, I thought GA did bots too. The bots/crawlers can't execute Javascript, Awstats would work for you since it reads the server logs. –  snacsnoc Jun 11 '09 at 2:37
    
I didn't know this, but Google Analytics actually records how many times it hits your site. That said, it doesn't count the OTHER bots, though. –  Pure.Krome Jun 29 '09 at 1:18

I use WebLog Expert when I want to examine our logs. There is a free version (lite) but that doesn't include the scheduler or command line functionality. The software itself runs as a desktop app (either remotely with ability to FTP logs or can be installed on the server), and can output reports in CSV, PDF and HTML.

share|improve this answer

why not outsourcing your web stats ? With solution like sitemeter, you just put an extra javascript, and they will give you stats on your site.

Web stats may be a pain for admin, doesn't always worth it to implement in house solution

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.