Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

What is the easiest way to get a count of the number of pages on a website? I don't want to actually download a local copy the entire site, just get a count of pages on it. Is there a tool (or combination of tools) that can crawl all the pages and links and give a total?

share|improve this question
up vote 3 down vote accepted

The quick and dirty way is to go to google and run a search like:

This example shows 232 known pages for

That will return the number of pages that google is aware of on that site. You may need to adjust your google preferences to include all content types (Turn SafeSearch off) and click the 'some results were omitted' warning before it'll give you its most accurate count.

To do it manually is harder. In order to discover all the pages on a particular website, you'll have to download the landing page, parse it for links that refer to the same web domain, then iteratively download those HTML pages and scan them as well. This continues iteratively until all links have been checked.

This method takes time (although with a tool like HTTrack, you can turn off non-HTML content downloading to save time).

This method will also miss orphaned pages that are not linked from the main page of the site.

share|improve this answer

Its old school, but it will work. Recursive calls to this should do the trick.

share|improve this answer
This link is broken! – dotoree Jul 4 '14 at 8:27

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.