Recommended system self test tool (what software should I use)?

Question

We have a setup of:

• ~20 nodes on EC2
• Several AppEngine deployments
• Several small locally hosted nodes

All these machines represented 1 distributed system that does "cool things".

The problem is it's also breaks in "cool ways", so we need to verify it's online and working.

We already use stuff like pingdom & custom automatic recovery if node simply dies, what is missing is a complete "System Wide", "Full cycle" test.

I'm looking for something that is designed to do just that. It should have the option to add custom tests. A test for example would be "Connect via HTTPS to foo.com/path/param1=7 and verify the result", or "Ensure that logs on node1, node2, node3, node4" are updating or maybe "Issue a service request to a predefined BIT pattern and make sure you get result in no less then Xms Oh and it should be GZip'ed as well". I'm assuming that such system would provide ability to add custom scripts to do all kinds of various tests and scenarios.

In the end (I think) it should present this in a nice web based GUI to show what tests passed, what has failed and for how long. This idea is to add unit tests for each new issue we discover so that we won't need to rediscover it's failing again (from the client!!).

I would like just to add that I'm not talking about stuff like Nagious or Monit but a more "full cycle testing system" So that it can breaks tests into several steps: Connect to A, wait 1 minute, connect to be and make sure you find the log of your connection to A. Then wait 2 hours and make sure analytics has counted your connection attempt and produced a valid result.

I should run on Linux, because scripting on Linux is oh so much better then the alternatives.

Thank you for reading,

Recommendations are highly appreciated.

Maxim.

Answer 1

Nagios is an excellent tool for this. It is primarily a scheduling engine - so you can schedule anything you can code - but there are lots of of probes available off the shelf and even tools for distributing probes to run on the monitored systems. On top of scheduling the probes you also get great control over notifications, escalations and automatic responses.

You've said that you think Nagios is not appropriate for "full life cycle" stuff - that's not the case. I've got a setup which runs complex scripted HTTP application checks, end-to-end email routing checks and all sorts of other things.

Answer 2

If the app you want to test is web only, you could try to use Selenium for that. Otherwise, you could also have a look at Cucumber which let's you specify functional tests. Not sure if any of these will give you as much flexibility as you wish though.

Answer 3

I have a feeling that AlertFox's hosted web transaction monitoring can take care of > 70% of your requirements.

E. g. it covers "Connect via HTTPS to foo.com/path/param1=7 and verify the result", or "Ensure that logs on node1, node2, node3, node4" are updating or maybe "Issue a service request to a predefined BIT pattern and make sure you get result in no less then Xms Oh and it should be GZip'ed as well" - as long as that can be checked via the web interface.

It will not cover very specific items like "Connect to A, wait 1 minute, .... Then wait 2 hours and ..."