1
vote

Is there any way to figure out how many bots (even if it's just one type of bot) that hits our site, on a daily, weekly and monthly average?

we have the standard IIS6 web site logs in lots of files.

Any tricks or recommendations?

Update

Program needs to be .NET based. No php or extra ugly packages, please. Free would be a bonus.

5 Answers 5

0
votes

Many reporting packages can report on this.

For instance, Awstats says

AWStats is able to detect robots visits.[...] "Robots visitors" are reported separately in the "Robots/Spiders visitors" chart.

EDIT: AWStats is free, written in Perl, and works just fine on Windows and with IIS logs.

4
  • I've updated my original post. Bascially, i can't use PHP. has to be .NET based ... and preferably free.
    – Pure.Krome
    May 21, 2009 at 7:34
  • AWStats is based on Perl, not PHP.
    – WerkkreW
    May 21, 2009 at 7:52
  • AWstats has only very basic bot-detection. Anything that has a non-robot useragent and does not request robots.txt is not recognised as a robot.
    – Jacco
    May 21, 2009 at 10:55
  • re: Php/Perl. opps. I ment Perl, but same thing.. i don't want to install extra stuff besides the app iteself.
    – Pure.Krome
    May 22, 2009 at 5:56
1
vote

Here is a pretty good list of known bot user-agents. I am not familiar with any free web log analytic tools written in .NET, but you could easily parse the logs yourself if all you are looking for are bots.

http://www.user-agents.org

1
vote

Google Analytics anyone? Free, Javascript based, so no need for server side languages. In depth coverage of hits, bots too.

3
  • I'm already using Google Analytics. It has stats on it's own robot hits?
    – Pure.Krome
    Jun 11, 2009 at 0:54
  • Hm, sorry I misled you, I thought GA did bots too. The bots/crawlers can't execute Javascript, Awstats would work for you since it reads the server logs.
    – snacsnoc
    Jun 11, 2009 at 2:37
  • I didn't know this, but Google Analytics actually records how many times it hits your site. That said, it doesn't count the OTHER bots, though.
    – Pure.Krome
    Jun 29, 2009 at 1:18
0
votes

I use WebLog Expert when I want to examine our logs. There is a free version (lite) but that doesn't include the scheduler or command line functionality. The software itself runs as a desktop app (either remotely with ability to FTP logs or can be installed on the server), and can output reports in CSV, PDF and HTML.

0
votes

why not outsourcing your web stats ? With solution like sitemeter, you just put an extra javascript, and they will give you stats on your site.

Web stats may be a pain for admin, doesn't always worth it to implement in house solution

Not the answer you're looking for? Browse other questions tagged .