1

I'm running Apache 2.2, and my server runs well. Noticed this weird anomaly in my access.log file, how should I prevent it? robots.txt doesn't seem to be working.

127.0.0.1 - - [17/Apr/2011:12:17:00 +0100] "GET / HTTP/1.1" 200 3022 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [17/Apr/2011:12:17:00 +0100] "GET /icons/blank.gif HTTP/1.1" 200 487 "http://localhost/" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [17/Apr/2011:12:17:00 +0100] "GET /icons/layout.gif HTTP/1.1" 200 616 "http://localhost/" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [17/Apr/2011:12:17:00 +0100] "GET /icons/folder.gif HTTP/1.1" 200 564 "http://localhost/" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [17/Apr/2011:12:17:00 +0100] "GET /icons/compressed.gif HTTP/1.1" 200 1379 "http://localhost/" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
127.0.0.1 - - [17/Apr/2011:12:17:01 +0100] "GET /icons/image2.gif HTTP/1.1" 200 650 "http://localhost/" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"

Is this a bot attack, or has my server screwed up? How should I fix it, to prevent this happening again?

1
  • Have you replaced the remote IP by 127.0.0.1 and your host by localhost? Or is Apache behind a proxy? It's really easy to spoof the User-Agent.
    – Lekensteyn
    Apr 18, 2011 at 13:16

2 Answers 2

1

Do you have robots.txt?

If not then you should create one. You can read about it here.

1
  • Added robots.txt - but it still shows up! Apr 17, 2011 at 22:56
1

MSNbot would be the webcrawler Microsoft uses for msn/live/bing. It's not "dangerous" normally. It will pickup robots.txt files and obey them if you have one correctly configured (see user74596's answer). It may take a day or two before it picks up the robots.txt file.

If you want people to be able to find your website in the search engines listed above, you shouldn't block access to your site.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.