We have a setup of:
- ~20 nodes on EC2
- Several AppEngine deployments
- Several small locally hosted nodes
All these machines represented 1 distributed system that does "cool things".
The problem is it's also breaks in "cool ways", so we need to verify it's online and working.
We already use stuff like pingdom & custom automatic recovery if node simply dies, what is missing is a complete "System Wide", "Full cycle" test.
I'm looking for something that is designed to do just that. It should have the option to add custom tests. A test for example would be "Connect via HTTPS to foo.com/path/param1=7 and verify the result", or "Ensure that logs on node1, node2, node3, node4" are updating or maybe "Issue a service request to a predefined BIT pattern and make sure you get result in no less then Xms Oh and it should be GZip'ed as well". I'm assuming that such system would provide ability to add custom scripts to do all kinds of various tests and scenarios.
In the end (I think) it should present this in a nice web based GUI to show what tests passed, what has failed and for how long. This idea is to add unit tests for each new issue we discover so that we won't need to rediscover it's failing again (from the client!!).
I would like just to add that I'm not talking about stuff like Nagious or Monit but a more "full cycle testing system" So that it can breaks tests into several steps: Connect to A, wait 1 minute, connect to be and make sure you find the log of your connection to A. Then wait 2 hours and make sure analytics has counted your connection attempt and produced a valid result.
I should run on Linux, because scripting on Linux is oh so much better then the alternatives.
Thank you for reading,
Recommendations are highly appreciated.