Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I curious how others monitor and detect crawler robots & IPs that are using up a lot of bandwidth on a webserver? This isn't for preventing DDoS attacks as much as bots run amok. What tools or processes do you use?

share

migration rejected from stackoverflow.com Aug 25 '13 at 21:37

This question came from our site for professional and enthusiast programmers. Votes, comments, and answers are locked due to the question being closed here, but it may be eligible for editing and reopening on the site where it originated.

closed as primarily opinion-based by Tom O'Connor Aug 25 '13 at 21:37

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise.If this question can be reworded to fit the rules in the help center, please edit the question.

6 Answers 6

Even though it's not a programming question, I'd have to say that doing a combination of grep, awk, sort, uniq; you can really come up with some good numbers from log files.

share

reverse the question: "Is the bandwidth high because we have a programming issue?"

and robots.txt isn't being respected?

share

One approach, done in php. http://www.bad-behavior.ioerror.us/

Not my work, but might be interesting to you...

share
    
searching through comments on the site it appears to block google crawlers at times. /shudder –  steve Feb 16 '09 at 15:06

Microsoft has a new extension for IIS 7 called Dynamic IP Restrictions Extension for IIS 7.0 - Beta.

"The Dynamic IP Restrictions for IIS 7.0 is a module that provides protection against denial of service and brute force attacks on web server and web sites. Such protection is provided by temporarily blocking IP addresses of the HTTP clients who make unusually high number of concurrent requests or who make large number of requests over small period of time." http://learn.iis.net/page.aspx/548/using-dynamic-ip-restrictions/

share

The question you raise is underdeveloped in OSS. Usually the line between abusive and normal activity is quite thin.

Bots and spiders are expected to obey robots.txt and doesn't touch parts that are denied (honeypots). If they do, they are added to blacklist filtered by firewall (either on permanent or on temporary basis).

Attacks however would have to raise many flags that after analysis can designate the requests as abusive. Best results are with statistical analysis approaches, with Baesian filtering being widely known. Human is training system to know what set of flags mean abusive activity and what not.

share

Use BRO + Snort.

Bro enhances snort with solid huristics and anomaly detection capabilities which, over time, increase in site-specific detection (i.e. it learn's what's normal for your site, and alerts to unusual activity).

-- from http://www.bro-ids.org/Features.html -- Technical Details Overview

Architecturally, Bro is layered into three major components. It uses the now-standard libpcap packet-capture library developed by LBNL's Network Research Group to filter the packet stream in the kernel for high performance.

The filtered packets are then passed along to Bro's event engine, which reduces the filtered stream into a series of higher-level events. These events reflect network activity in policy-neutral terms. That is, what has happened but not why or whether it is significant.

Finally, Bro's policy script interpreter executes event handlers written in the custom Bro scripting language. These scripts express a site's security policy, i.e., what actions to take when the monitor detects different types of activity. The scripting language is very powerful, allowing Bro to track detailed information about the network's activity. Scripts can generate real-time alerts and can also execute arbitrary programs. This last means that Bro can respond to attacks. For example, at LBNL the operational Bro's configuration will sometimes block a remote Internet host from further access to the Laboratory's network, which can be viewed as a form of reactive firewall.

share