Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

My logwatch report which I installed recently shows me this:

--------------------- httpd Begin ------------------------
0.78 MB transferred in 5864 responses  (1xx 0, 2xx 4900, 3xx 0, 4xx 964, 5xx 0)
160 Images (0.16 MB),
857 Content pages (0.62 MB),
4847 Other (0.00 MB)

Requests with error response codes
404 Not Found
  /%E2%80%98planeat%E2%80%99-film-explores-l ... greenfudge-org/: 1 Time(s)
  /10-foods-to-add-to-the-brain-diet-to-help ... -function/feed/: 1 Time(s)
  /10-ways-to-reboot-your-body-with-healthy- ... s-and-exercise/: 1 Time(s)
  /bachmann-holds-her-ground-against-raising ... com-blogs/feed/: 1 Time(s)
  /behind-conan-the-barbarians-diet/: 1 Time(s)
  /tag/dietitian/: 1 Time(s)
  /tag/diets/page/10/: 1 Time(s)
  /tag/directory-products/feed/: 1 Time(s)
  /wp-content/uploads/2011/06/1309268736-49.jpg: 1 Time(s)
  /wp-content/uploads/2011/06/1309271430-30.jpg: 1 Time(s)
  /wp-content/uploads/2011/06/1309339847-35.jpg: 1 Time(s)

my note here: there are really a lot of these kind of requests like above and I pasted just a few because of clarity.

  A total of 12 ROBOTS were logged
  Mozilla/5.0 (compatible; Googlebot/2.1; + 2 Time(s)
  Mozilla/5.0 (compatible; YandexBot/3.0; + 5 Time(s)
  Twitterbot/1.0 1 Time(s)
  Mozilla/5.0 (compatible; AhrefsBot/2.0; + 4 Time(s)
  Sosospider+(+ 3 Time(s)
  msnbot/2.0b (+ 1 Time(s)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.2; 1    Time(s)
  msnbot-media/1.1 (+ 77 Time(s)
  Mozilla/5.0 (compatible; Ezooms/1.0; 1 Time(s)
  Mozilla/5.0 (compatible; bingbot/2.0; + 17 Time(s)
  Baiduspider+(+ 11 Time(s)
  Mozilla/5.0 (compatible; Butterfly/1.0; +    Gecko/2009032608 Firefox/3.0.8 1 Time(s)
  ---------------------- httpd End -------------------------

So, I'm thinking this is somekind of a bot (and potentialy one of the listed ones above), so can you please direct me on how could I prevent them from guessing the links in hope of finding content?

edit: since i own a VPS server, there are a lot of domains on it. Can you tell me how can I know on which domain particular 404 happened? Like this line for example: /tag/dietitian/

share|improve this question
up vote 7 down vote accepted

You don't, really, anymore than you can stop regular users guessing at links. Correctly secure your content and this won't be an issue anyway.

Obscure links are not a safe way to hide things.

You can ensure you've got a correctly configured robots.txt - that'll stop most of the legit bots.

share|improve this answer
@Nikola Have you tried Google? Server Fault is for professional only, and professionals are expected to be able to do at least an absolutely minimal degree of research on their own. – Chris S Mar 7 '12 at 14:10

One way would be to use fail2ban and configure it to match your needs. In short: among the other its features, fail2ban can tail your Apache access log and after X amount of Y kind of matches can put the accessing client to a Z minutes penalty by blocking the client IP for XX minutes.

Usually enough to scare the bots away, but beware, this can very well block legitimate users if you are not careful enough.

share|improve this answer

Search engine crawlers don't guess links - they just follow them unless dissuaded by a nofollow or robots.txt rule.

If you have requests for things that don't exist from a search engine's bot - the crawler is following a link on a publicly accessible page that points at it, the correct action is to correct/remove the reference.

If it's a malicious bot - all you can do is detect it and block access. If the bot is announcing itself - that's easy you could for example block with a rewrite rule

share|improve this answer

You can add Virtualhost info to the log, or use separate logs per VirtualHost

See Apache Log Files Documentation

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.