Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

What is the easiest way to get a count of the number of pages on a website? I don't want to actually download a local copy the entire site, just get a count of pages on it. Is there a tool (or combination of tools) that can crawl all the pages and links and give a total?

share|improve this question

2 Answers 2

up vote 3 down vote accepted

The quick and dirty way is to go to google and run a search like:

site:mydomain.com

This example shows 232 known pages for fronde.com: http://i47.tinypic.com/j0h003.jpg

That will return the number of pages that google is aware of on that site. You may need to adjust your google preferences to include all content types (Turn SafeSearch off) and click the 'some results were omitted' warning before it'll give you its most accurate count.

To do it manually is harder. In order to discover all the pages on a particular website, you'll have to download the landing page, parse it for links that refer to the same web domain, then iteratively download those HTML pages and scan them as well. This continues iteratively until all links have been checked.

This method takes time (although with a tool like HTTrack, you can turn off non-HTML content downloading to save time).

This method will also miss orphaned pages that are not linked from the main page of the site.

share|improve this answer

Its old school, but it will work. Recursive calls to this should do the trick. http://search.cpan.org/dist/HTML-Parser/lib/HTML/LinkExtor.pm

share|improve this answer
    
This link is broken! –  dotoree Jul 4 at 8:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.