Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I want to block some bad search engine bots like MJ12bot, YandexBot and Ezooms. We have like 200 users in the directadmin environment, and we want to install a "plugin" or "mod" to block those request because we dont't want to go to every site to add it to the .htaccess.

Is there a plugin or mod for this for directadmin, and what is it called?

Thanks!

share|improve this question
    
You may get better answers at our sister site Pro Webmasters. Be sure to search for existing questions before asking, as this topic is very commonly discussed there. –  Michael Hampton Dec 18 '12 at 17:33
add comment

closed as off topic by Michael Hampton, mdpc, Ward, Scott Pack, Bryan Dec 18 '12 at 20:41

Questions on Server Fault are expected to relate to professional server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

1 Answer

up vote 1 down vote accepted

Create robots.txt file with the following contents under (/var/www/html):

User-agent: YandexBot
User-agent: Ezooms
Disallow: /

Add the following to your httpd.conf file

Alias /robots.txt /var/www/html/robots.txt

This robots.txt file will now be served for all virtual hosts on your server, overriding any robots.txt file you might have for individual hosts.

share|improve this answer
    
but if i want a custom robots.txt, this will override them? Also, i have the feeling that this kind of bots not abay the robots.txt, thats why i want to block them completely. –  Michel Bardelmeijer Dec 18 '12 at 13:14
    
@MichelBardelmeijer If the bots don't obey robots.txt your only option is IP/Subnet/AS level blocking (fine-grained, maintenance-intensive --> blocking huge swaths of the internet, but not much maintenance). –  voretaq7 Dec 18 '12 at 18:00
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.