Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We were a little surprised to see this on our Cacti graphs for June 4 web traffic:

We ran Log Parser on our IIS logs and it turns out this was a perfect storm of Yahoo and Google bots indexing us.. in that 3 hour period, we saw 287k hits from 3 different google ips, plus 104k from yahoo. Ouch?

While we don't want to block Google or Yahoo, this has come up before. We have access to a Cisco PIX 515E, and we're thinking about putting that in front so we can dynamically deal with bandwidth offenders without touching our web servers directly.

But is that the best solution? I'm wondering if there is any software or hardware that can help us identify and block excessive bandwidth use, ideally in real time? Perhaps some bit of hardware or open-source software we can put in front of our web servers?

We are mostly a windows shop but we have some linux skills as well; we're also open to buying hardware if the PIX 515E isn't sufficient. What would you recommend?

share|improve this question
    
Thanks, I added that image –  Jeff Atwood Jun 16 '12 at 18:39
add comment

8 Answers

up vote 21 down vote accepted

If your PIX is running version 7.2 or greater of the OS, or can be upgraded to it, then you can implement QOS policies at the firewall level. In particular this allows you to shape traffic and should allow you to limit the bandwidth used by bots. Cisco have a good gudie to this here.

share|improve this answer
    
The only issue I have with this answer is that the poster doesn't want to BLOCK (or degrade) Google or Yahoo but rather wants them to "crawl nice". –  KPWINC Jun 5 '09 at 21:53
1  
QOS is not about blocking. That's why Sam Cogan wrote "should allow you to limit the bandwidth used by bots". –  Piotr Dobrogost Jun 6 '09 at 18:02
add comment

I'm not sure about yahoo, but you can configure the frequency Google's bot indexes your site. Have a look at Google Webmasters. I'm not sure if Yahoo has anything similar. At any that'll reduce your traffic up to 50%.

Alternatively, some web servers can limit traffic per connection so you can try that. I personally would stay away from hardware solutions since it's most likely going to cost more.

share|improve this answer
    
Yahoo! has Site Explorer. –  Amuck Oct 30 '09 at 15:35
add comment

To reduce the crawling load - This only works with Microsoft and Yahoo. For Google you’ll need to specify a slower crawling speed through their Webmaster Tools (http://www.google.com/webmasters/).

Be VERY careful when implementing this because if you slow down the crawl too much, robots won’t be able to get to all of your site and you may lose pages from the index.

Here are some examples (these go in your robots.txt file)

# Yahoo's Slurp Robot - Please wait 7 seconds in between visits

User-agent: slurp
Crawl-delay: 7

# MSN Robot - Please wait 5 seconds in between visits

User-agent: msnbot
Crawl-delay: 5

Slightly off topic but you can also specify a Sitemap or Sitemap index file.

If you’d like to provide search engines with a comprehensive list of your best URLs. You can also provide one or more Sitemap autodiscovery directives. Please note that user-agent does not apply to this directive so you cannot use this to specify a Sitemap to some but not all search engines.

# Please read my sitemap and index everything!

Sitemap: http://yourdomain.com/sitemap.axd
share|improve this answer
add comment

We use a Watchguard firewall (ours is a X1000 which is end-of life now). They have many feautres revolving around blocking domains or ips who are seen time and time again or are using an obsesive amount of bandwidth.

This would need some tweaking because you obvisouly would not want to block Jon Skeet on stackoverflow :)

share|improve this answer
add comment

I'd recommend Microsoft ISA Server 2006. Specifically for this requirement, it will limit to 600 HTTP requests/min per IP by default and you can apply an exception for Jon Skeet (sorry, I realise that "joke" has been made already!).

You have the additional benefits of application-level filtering, the ability to load-balance across multiple webservers (instead of NLB on those servers), VPN termination etc. There's a number of commercial extensions available and you can even write your own ISAPI filter if you're feeling brave.

It's obviously not open-source but has benefits to a Windows shop and runs on commodity hardware.

share|improve this answer
    
This is the most flexible approach along with modifyig robots.txt –  Jim B Jun 6 '09 at 5:20
    
Thanks. I agree that the crawlers need to be given some direction! –  Robin M Jun 6 '09 at 7:02
add comment

We use Foundry load-balancers (specifically SI850s) to handle this kind of shaping issue, it also handles quite a lot of other 'nastys' like SYN-floods etc. Might be a bit overkill for you guys though.

share|improve this answer
add comment

Bluecoat (formerly Packeteer) PacketShaper products can dynamically throttle excessive usage on traffic that it manages.

share|improve this answer
add comment

You can even perform rudimentary rate-limiting with any regular Cisco router of any decent capacity/vintage. Are you using a Cisco router?

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.