How should I check a site for dead links? [closed]

Question

How should I check my website for dead links? I've used some web based utilities that were okay, e.g.:

• http://dead-links.com
• http://validator.w3.org/checklink

I've used some freeware apps in the past, like Xenu.

I'd really like something that could export reports in a format I could do something with, like CSV or XML. What do you use?

Answer 1

The IIS SEO toolkit is great for this. It does a lot more than just search for broken links. http://www.iis.net/extensions/SEOToolkit

you can run reports with the data and also track reports over time.

Answer 2

I have a Linux machine that has a cron job that runs linkchecker to send me a report.

http://wummel.github.io/linkchecker/

If you are running Ubuntu it is in the package manager.

sudo aptitude install linkchecker
man linkchecker

Lots of options. Works well for me. Can save the report in various formats.

Answer 3

I'll vote for Xenu. Blindingly fast and gives you all kinds of other features.

Answer 4

I haven't tried this, but I came across it last night while I was trying to beat wget into doing something else. May or may not be helpful in your case.

   --spider
       When invoked with this option, Wget will behave as a Web spider,
       which means that it will not download the pages, just check that
       they are there.  For example, you can use Wget to check your book‐
       marks:  

               wget --spider --force-html -i bookmarks.html

       This feature needs much more work for Wget to get close to the
       functionality of real web spiders.

Answer 5

I'd personally go with some server-side log analyzers for continuous monitoring of dead links. Webalizer/Awstats and probably other tools can give you a report of any 404 Not Found errors by parsing your webserver log.

Advantage of this approach: You also catch incorrect links to your site which might be posted on some other site, user typos and so on. And it's continuous monitoring, not just one time when you run some dead link checker.

You can combine this with a website mirroring software like httrack. Just have it crawl your site and then get the report from one of the log parsing tools.

There are also some desktop apps which can parse apache log files and give you any 404 errors in a nice report.