Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

we have a situation where we log visits and visitors on page hits and bots are clogging up our database. We can't use captcha or other techniques like that because this is before we even ask for human input, basically we are logging page hits and we would like to only log page hits by humans.

Is there a list of known bot IP out there? Does checking known bot user-agents work?

share|improve this question
    
Related stackoverflow.com/questions/2776013/…. –  Bratch May 5 '10 at 20:07

3 Answers 3

up vote 1 down vote accepted

There are a few elements to pursue with this.

user-agent string is one value, but it can be trivially spoofed.

I've found a reasonably useful heuristic is to do a bit of pre-processing, then look at traffic:

  • Reverse IP lookups
  • Invoke the Routeviews project (http://www.routeviews.org/) to pull up CIDR and ASN for a given IP. This allows you to expand from single IPs to contiguous network ranges with similar traffic patterns.
  • Pull down ASN names: http://bgp.potaroo.net/cidr/autnums.html

Parse out your access logs adding host, ASN, CIDR, and ASN name information. Subset URLs to the nonvariant part (stripping everything past '?' generally, though YMMV). If you've got specific search or utility pages, focus on these (typically I've seen problems either with bots using some sort of user verification service, or search).

Look for single IPs with high volumes of traffic.

Look for single CIDR blocks or ASNs with high volumes of traffic.

Rule out legitimate search traffic (Google, Bing, Yahoo, Baidu, Facebook, and similar bots / network space). This is probably going to be one of your larger areas of ongoing maintenance, this stuff changes all the time.

Rule out legitimate user traffic. Especially for high-volume users of your site.

Identify what normal patterns of usage are, for both end-users and search bots. If a typical user visits 1-3 pages per minute, with a typical session of 5-10 minutes, and Googlebot limits itself to, say, 10 searches per minute, and you suddenly see a single IP or CIDR block lighting up with hundreds or thousands of searches per minute, you may have found your problem.

Investigate the origins of high-volume / high-impact (in a negative sense) traffic. Frequently a WHOIS query will reveal that this is some sort of hosting space -- not typically where you'll see a lot of legitimate user traffic. Patterns may appear in user-agent strings, request URLs, referrer strings, etc., that tip you off to additional patterns.

A caching whois client can be a big assistance if you end up doing a lot of WHOIS lookups, both the speed the process, and to avoid rate-limiting/throttling by registrars (for some reason, they don't take kindly to entities conducting thousands of repeat/automated lookups). You may be able to go to registrars directly for more information, though I haven't pursued this.

Checks against various reputation databases (spam lookups, SenderBase, there's now some Google stuff along these lines) may also corroborate poorly-policed network space.

I'd love to say I've got something to sell you along these lines, but what I'm working with is mostly some awk and other tools to pull this together. It'll parse a million lines of log a minute or so (plus a bit of preparatory overhead to prepare hashes for IPs and ASN/CIDR information) on a modest workstation. Not fully automated, but it'll give me a decent picture of an issue with a few minutes of work.

share|improve this answer

I would think from the sysadmin standpoint repeated hits from a single IP at a regular interval would indicate a likely bot. You could find this by simply parsing the logs.

I might first filter out IPs with a large number of hits. Then fill an array with the times of those hits, and maybe look at standard deviation of the interval between those hits.

The distinct advantage to a solution like this is that get to write something fairly interesting if you are working a full time admin ;-)

share|improve this answer

A quick google search revealed this site. It could be a good starting point.

Checking only for the user agent may not be enough since the user agent can be easily forged.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.