I am using an application called Xenu Link Sleuth to try and find broken links on a site we host. When I go to the site through a browser it pops right open.
When I try to run it through Xenu it immediately throws a 404 not found error. I checked the Robots.txt file thinking that maybe Xenu was using that as a base but I determined that was not the cause.
Does anyone know of anything that could case this? Could there be security setup somewhere that I don't know about that throws a 404 for crawlers?
Any help appreciated, i'm stumped.
Also, this is an intranet site. Not sure if that makes a difference.