0

I am using an application called Xenu Link Sleuth to try and find broken links on a site we host. When I go to the site through a browser it pops right open.

When I try to run it through Xenu it immediately throws a 404 not found error. I checked the Robots.txt file thinking that maybe Xenu was using that as a base but I determined that was not the cause.

Does anyone know of anything that could case this? Could there be security setup somewhere that I don't know about that throws a 404 for crawlers?

Any help appreciated, i'm stumped.

Also, this is an intranet site. Not sure if that makes a difference.

2 Answers 2

0

I would check a couple of things… First make sure the URL isn't breaking it. There were all sorts of articles earlier this week about hashbangs #! breaking things. Also, I would look at what your request looks like in the logs vs what this softwares request looks like in the logs.

1

404 error is thrown when a website page is not find. Please make sure that you set the default document for the site.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .