Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am using an application called Xenu Link Sleuth to try and find broken links on a site we host. When I go to the site through a browser it pops right open.

When I try to run it through Xenu it immediately throws a 404 not found error. I checked the Robots.txt file thinking that maybe Xenu was using that as a base but I determined that was not the cause.

Does anyone know of anything that could case this? Could there be security setup somewhere that I don't know about that throws a 404 for crawlers?

Any help appreciated, i'm stumped.

Also, this is an intranet site. Not sure if that makes a difference.

share|improve this question
up vote 0 down vote accepted

I would check a couple of things… First make sure the URL isn't breaking it. There were all sorts of articles earlier this week about hashbangs #! breaking things. Also, I would look at what your request looks like in the logs vs what this softwares request looks like in the logs.

share|improve this answer

404 error is thrown when a website page is not find. Please make sure that you set the default document for the site.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.