I used JMeter, Zabbix and Splunk to do this, there's a blog entry on how I set it up http://engineering.navinet.net/?p=322. I'll explain briefly how it all works.
Essentially, we used JMeter to record the various HTTP Requests required to carry out the various transactions/processes in our application and then run these with dummy users. As part of this test plan we created our own Beanshell Sampler script which got executed after each request. This script created Sys Log format messages with details on the URL hit, status code, timing etc. as well as the Header variables. Internally we use Splunk for storing and querying machine data such as event logs etc. so we used the Sampler script to send the data into Splunk.
Using the functionality within Splunk we were able to create visual dashboards and charts reporting on the durations for each execution and show error counts etc. We also set up some reports which were ran on a regular basis which emailed when either:
- The time to execute a step took longer than a defined threshold (based on an average over defined window)
- No data was found for certain steps within a given time frame
We then used Zabbix to kick off the JMeter test plans using it's User Parameter functionality on a regular basis and had it track when errors occured during execution. This would then notify us when error's occured during execution and we would be able to debug using Splunk.
If you're interested I can forward on some screenshots or scripts to help people get started.