Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I was having a look at DRKSpider to find problems with a website in our production server, but it seems its export feature generates different outputs (with different content).

My goal is to find a good tool that shows every type of status code that might be an error 404, 500, 403, etc.

Could you guys please suggest some open source tools to craw a website in order to list all server codes related to error?

share|improve this question

closed as off-topic by masegaloeh, Joel Coel, Andrew Schulman, HBruijn, Jenny D Jun 22 '15 at 9:18

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

man wget and it's spider option. Or HTTrack or cURL or anything else found on Google. Even Google itself with the Webmaster tools. – mailq Aug 15 '11 at 18:26
up vote 0 down vote accepted

I think the hardest part of this is that most open source tools won't implement a full DOM with a js and css engine. So you're likely to run into issues where even using wget will not expose broken java script issues on your site. If you are trying to figure out what errors your site might be generating for users you should look at implementing a spider that does support js/css/etc. Something like: (free up to 10,000 pages)

You can also use google webmaster tools like @mailq mentioned, here is more details on their crawl errors section:

Lastly, if you aren't already you should be watching your logs for these errors and tracking referrer information so you can hopefully investigate them then as well.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.