|
I was having a look at DRKSpider to find problems with a website in our production server, but it seems its export feature generates different outputs (with different content). My goal is to find a good tool that shows every type of status code that might be an error 404, 500, 403, etc... Could you guys please suggest some open source tools to craw a website in order to list all server codes related to error? thank you
| ||||
|
feedback
|
|
I think the hardest part of this is that most open source tools won't implement a full DOM with a js and css engine. So you're likely to run into issues where even using wget will not expose broken java script issues on your site. If you are trying to figure out what errors your site might be generating for users you should look at implementing a spider that does support js/css/etc. Something like: http://atomz.com/ (free up to 10,000 pages) You can also use google webmaster tools like @mailq mentioned, here is more details on their crawl errors section: Lastly, if you aren't already you should be watching your logs for these errors and tracking referrer information so you can hopefully investigate them then as well. | |||
|
feedback
|
man wgetand it's spider option. Or HTTrack or cURL or anything else found on Google. Even Google itself with the Webmaster tools. – mailq Aug 15 '11 at 18:26