Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've perused through the forums here and was wondering what the best practice was in terms of monitoring the status of your "entire" website? (not just the homepage) In terms of priority, if your homepage is down, typically your entire site is down. But at a much more granular level, I would like to know how to monitor if a section or page of our site is not responding.

We don't have a budget for high-end tools like webmetrics or gomez. I've looked at pingdom and bello and both tools seems to just check if your homepage is up and running.

We are a windows shop, that host a 300,000+ page website. I understand its not realistic to monitor each page. Would it make sense to just run a script to check if the main pages on the site are up? For example, if we check if the main apparel page is up, then we can assume its sub-pages (mens clothing, womens clothing) are also up.

Any advice would be greatly appreciated. Thanks!

share|improve this question
"I understand its not realistic to monitor each page." -- There are tools which will monitor all network traffic going in and out through the webserver (Often using a spanning port on your network switch). They can observe a tremendous amount of information, and will see any 4xx errors or 3xx responses for any pages. It's like an IDS, but focused on HTTP traffic. However, I think these are generally in the $10,000+ range. – Stefan Lasiewski Jun 22 '10 at 22:26
Please be aware that this is not a forum, it is a Q&A site. Please read the FAQ. – John Gardeniers Jun 22 '10 at 22:55
I apologize, in my haste, I typed forum, but meant Q&A site. – JoefrshnJoeclean Jun 23 '10 at 17:47
up vote 4 down vote accepted

You could use Nagios. There is a check_http plugin that you could use multiple times. Give one instance a url of http://myserver/, another of http://myserver/apparel, a third of http://myserver/hardware, etc. Also give one an instance of to check FQDN usage.

From the check_http documentation: This plugin will attempt to open an HTTP connection with the host. Successful connects return STATE_OK, refusals and timeouts return STATE_CRITICAL other errors return STATE_UNKNOWN. Successful connects, but incorrect reponse messages from the host result in STATE_WARNING return values.

share|improve this answer

In Apache, and I'm sure in IIS (or Apache under Windows), you can trap the 5xx series error pages to your own handler. If you are running varnish/squid/pound in front, you can set up a fallback host that would display a particular page on a failure. That fallback host could be a very simple, small stack webserver that would do nothing but set a status of a page you've monitored. The 5xx handler could intercept and log that data somewhere as well.

I'm not a real fan of the 5xx error generating emails/paging due to the fact that the database server might be down, and every one of your 300k pages might instantly start generating errors. However, intercepting the handler and using that to write/modify a page or status that is monitored would save you the hassle of spidering a huge sample of your pages.

share|improve this answer

While it might not be the "cleanest" solution, could you simply write a single (status) page that checks all of the various tiers w/in your system? If this page returns anything but some sort of success message, it means that there is a failure somewhere in the tier and you could be alerted.

Again, don't claim that this is a silver bullet - or even the most robust solution - but if you need something, and need it quick, this might do.

share|improve this answer

Both Kevin and Malonso solutions sound good to be combined...

Malonso solution would be like some kind of 'unit testint' that tests all logic, conections, permisions and so, and give an OK or a FAIL. That, plus, as Kevin says, pulling some critical webpages and see that eveything goes ok...

Because pages can load correctly but not working fine. Maybe pages load fine, but database stopped and some content is empty, or some modules aren't working at all and some content goes bad...

So I would write some tests on php, or asp or whatever you are using, and also pull pages with NAGIOS or something similar...

share|improve this answer

One idea is to use the SEO Toolkit ( which will crawl every page on your site and report the status code and the content as well.
This will not only give you nice reports you can later analyze, but you can even script it and monitor the number of 404's or 500's your server returns so that you can know which pages are having issues.

I wrote a blog to show how you can do that easily from a command line that could be added as a scheduled task:

share|improve this answer

You could try for external monitoring, similar to webmetrics and gomez but more intuitive and several valuable features.

You could also look at for mining your logs, but this is not going to tell you if your users cannot access your site.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.