Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We experience a lot of traffic and server load on a web server.

All I can find out is majestic12 accessing pages all the time.

I wonder how I can prevent majestic12 from indexing the site

Do they respect any robots.txt entry and how do I write such an enty?

share|improve this question
add comment

2 Answers

up vote 3 down vote accepted

According to Majestic 12's own page about robots.txt, they fully respect robots exclusion (3rd answer from the top). The robots.txt file is a plain text file in the root of your website, i.e. you place it at:

http://www.yourdomain.com/robots.txt

and have these lines in the file:

User-agent: MJ12bot
Disallow: /

So if you want to block that bot, I see no problem -- unless you're getting hammered by one of the fake bots they mention.

share|improve this answer
    
Thank you. Somehow I didn't find the faq on their pages but now i have it. I wonder if they have mentioned all fake bots. Why would someone claim to be majestic12? –  user12096 May 16 '10 at 9:30
add comment

For the op's follow up question:

I wonder if they have mentioned all fake bots. Why would someone claim to be majestic12?

That would be a false flag operation. Virus disguises itself as a legit bot/process to crawl ip's. The explanation is on the majestic12 faq way down.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.