Multi-stage CI monitoring or how can I split the deployment steps into controllable chunks

Question

First of all, I am not looking for a product recommendation (well, I am, but it's more like me looking for a solution to a problem and not trying to pick something from an existing list)

The environment: I am using Jenkins to run a complex, multi-stage jobs. The job creates and provisions servers, deploys the OS and the application stack on them, populates the stack with demo data, runs standard tests, then some additional negative tests, then scales the stack with some more provisioned servers and runs some more tests.

The problem: At any one of these stages a failure might occur, either just a reporter misconfiguration or a complete failure for a step. Either way, I can do one of two things in Jenkins - exit(1) or keep going, dumping the problem to the console. Both aren't great because I will end up with either a broken incomplete job (if I exit) or with a nice green false positive, where in order to make sure it's actually green I have to read a few megs of log dumps from the console.

What I'm looking for: A way for Jenkins or some other system, to be able to monitor and report the success/failure of each step in a job separately, maintaining the job consistency (e.g. I cannot start a new job for each step), so that I can see, for example, that provisioning and OS deployment worked, stack deployed with errors but completed, and negative tests failed, instead of the current green/red status per run with no detail unless I go in and read the logs

Any suggestions are welcome, whether it's for a plugin for Jenkins I'm not aware of, or to another system, or whatever else.

Answer 1

Are you using the pipeline as code for your Jenkins jobs or a traditional freestyle jobs linked together? If the latter, then while I know that you can set the job to proceed, I'm sure you can set it to email you when there is a failure right?

If you decide to use the former, you could achieve this with try and catch block. I have some steps which may or may not fail, and they are not a showstopper. So, I enclose the step in a try block and in the catch, I'll force set the build to success and send an email. It goes like this:

try {
  sh 'exit 1' //the step that might trigger a complete failure
}

catch(err) {
  currentBuild.result = 'SUCCESS' //force set the job to a success. i.e continue
  emailext ....   // email that the step failed
}