0

We supply Magento and Typo3 installations to customers. To improve QA we wanted to use an automatic link checker to check for broken and/or outdated links - automatically. We want to check all links staying inside it's own domain, and maybe links going out, but not crawl external domains.

However, due to the nature of shops, the amount of links can be a little on the larger side.

So far, we've used linkchecker (Linux script) and Xenu (Windows GUI Tool):

  • Linkchecker crawled to almost a halt over the course of 4 days without reaching a conclusion.
  • Xenu was blazingly fast at first, but it being a 32bit application, it stopped after having crawled about 300k links (of 1.4m) with mentioning that it went out of memory.

I understand that the already crawled URLs need to be stored to check if a new URL equals an old one and doesn't need a re-check.

Is there a tool, a website or any other solution that can crawl websites of that magnitude and present some kind of result (only broken links would be nice)?

Improve this question
1
  • Links <> unique Page URLs. (If you really mean links, 1.4 million is not that much.) Anyhow, you may want to try A1 Website Analyzer if you can not get Xenu working (maybe ask in the usergroup for it?)
    – Tom
    Commented Apr 14, 2014 at 22:25

1 Answer 1

Reset to default
0

Dabu, did you try to crawl your site directory by directory? If I remember right, Xenu provides such feature.

As an option, you can try to use our Site Visualizer web crawler for your task. I'm not sure if it will be able to crawl all of 1.4m links, since we haven't tried it on such large websites. It also allows to crawl only certain directories.

Download trial version and install it, then create new project and uncheck the following options of the Crawling tab in order to increase speed and minimize occupied resources:

  • Count Word Number
  • < IMG >
  • < LINK >
  • < SCRIPT >
  • Thread Number set to 25
  • URL Access Timeout set to 10

You can also define a certain directory (or directories) to crawl with Include URLs project option. For instance, you can crawl just your QA directory, since it contains the most of broken links. In the Include URLs dialog box type '/QA/' (w/o quotes) and click OK.

Broken links after the crawling is done, double click Client Error Responses (4xx) report on the Reports tab of the Site Visualizer main window.

Trial version of the app is full-featured, so you can use it for 30 days for free.

Improve this answer
2
  • That sounds like a good idea, thank you. I will look into the mentioned software. However, I could not find a functionality in Xenu that mentions crawling the FTP directory, and I'm not sure it would work, given the dynamic nature of the content.
    – Dabu
    Commented Feb 13, 2014 at 12:19
  • Please do not hesitate to contact Site Visualizer support if you won't be able to crawl your site. We're interested in solving your problem, since such large website crawling is a good test for our software product.
    – Oleg
    Commented Feb 13, 2014 at 12:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .