Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I'm running Apache 2.2, and my server runs well. Noticed this weird anomaly in my access.log file, how should I prevent it? robots.txt doesn't seem to be working.

127.0.0.1 - - [17/Apr/2011:12:17:00 +0100] "GET / HTTP/1.1" 200 3022 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [17/Apr/2011:12:17:00 +0100] "GET /icons/blank.gif HTTP/1.1" 200 487 "http://localhost/" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [17/Apr/2011:12:17:00 +0100] "GET /icons/layout.gif HTTP/1.1" 200 616 "http://localhost/" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [17/Apr/2011:12:17:00 +0100] "GET /icons/folder.gif HTTP/1.1" 200 564 "http://localhost/" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [17/Apr/2011:12:17:00 +0100] "GET /icons/compressed.gif HTTP/1.1" 200 1379 "http://localhost/" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [17/Apr/2011:12:17:01 +0100] "GET /icons/image2.gif HTTP/1.1" 200 650 "http://localhost/" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"

Is this a bot attack, or has my server screwed up? How should I fix it, to prevent this happening again?

share|improve this question
    
Have you replaced the remote IP by 127.0.0.1 and your host by localhost? Or is Apache behind a proxy? It's really easy to spoof the User-Agent. –  Lekensteyn Apr 18 '11 at 13:16

2 Answers 2

Do you have robots.txt?

If not then you should create one. You can read about it here.

share|improve this answer
    
Added robots.txt - but it still shows up! –  subarufan86 Apr 17 '11 at 22:56

MSNbot would be the webcrawler Microsoft uses for msn/live/bing. It's not "dangerous" normally. It will pickup robots.txt files and obey them if you have one correctly configured (see user74596's answer). It may take a day or two before it picks up the robots.txt file.

If you want people to be able to find your website in the search engines listed above, you shouldn't block access to your site.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.