0

We know that we can block certain spiders from crawling websites pages using robots.txt or .htaccess or maybe via the Apache configuration File httpd.conf.

But that requires to edit may be a large number of sites on some dedicated servers and bots still will "access" and consume the resources. Is there any other "safe" ways to block these IMG spiders from the root on servers like windows servers?

1 Answer 1

3

robots.txt doesn't block anything, it is up to the crawler whether it pays attention to robots.txt or ignores it. There's also no central list of web crawlers, since anyone can run one for any reason and they can appear as ordinary browsing traffic, claiming to come from an ordinary web browser.

You can do basic referrer checks to block image hotlinking, you can do intrusion prevention to block port scanners and malicious requests, but if want to block spiders and not people and not false alarms, you probably need to put your site behind a login page.

bots still will "access" and consume the resources.

Minimal resources. You'll spend hours implementing, testing and fiddling with a "spider blocking" policy, and you could spend that investment on hardware that can cope with it instead. It ought to be background noise, really.

If spiders are hammering your site, how will it cope with actual users?

2
  • Thought minimizing and optimizing everything could help in performance etc. Attacks, unwanted bots and anything that consume the resources for no reason. That was only a thought when I monitored the network and CPU from server Performance Monitor when those IPs connected. first time dedicated ;)
    – hsobhy
    May 1, 2013 at 2:31
  • How do you know what are "unwanted bots"? Do you not want your site to appear in search engines? May 1, 2013 at 2:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .