0

First of all, I am not looking for a product recommendation (well, I am, but it's more like me looking for a solution to a problem and not trying to pick something from an existing list)

The environment: I am using Jenkins to run a complex, multi-stage jobs. The job creates and provisions servers, deploys the OS and the application stack on them, populates the stack with demo data, runs standard tests, then some additional negative tests, then scales the stack with some more provisioned servers and runs some more tests.

The problem: At any one of these stages a failure might occur, either just a reporter misconfiguration or a complete failure for a step. Either way, I can do one of two things in Jenkins - exit(1) or keep going, dumping the problem to the console. Both aren't great because I will end up with either a broken incomplete job (if I exit) or with a nice green false positive, where in order to make sure it's actually green I have to read a few megs of log dumps from the console.

What I'm looking for: A way for Jenkins or some other system, to be able to monitor and report the success/failure of each step in a job separately, maintaining the job consistency (e.g. I cannot start a new job for each step), so that I can see, for example, that provisioning and OS deployment worked, stack deployed with errors but completed, and negative tests failed, instead of the current green/red status per run with no detail unless I go in and read the logs

Any suggestions are welcome, whether it's for a plugin for Jenkins I'm not aware of, or to another system, or whatever else.

1

Are you using the pipeline as code for your Jenkins jobs or a traditional freestyle jobs linked together? If the latter, then while I know that you can set the job to proceed, I'm sure you can set it to email you when there is a failure right?

If you decide to use the former, you could achieve this with try and catch block. I have some steps which may or may not fail, and they are not a showstopper. So, I enclose the step in a try block and in the catch, I'll force set the build to success and send an email. It goes like this:

try {
  sh 'exit 1' //the step that might trigger a complete failure
}

catch(err) {
  currentBuild.result = 'SUCCESS' //force set the job to a success. i.e continue
  emailext ....   // email that the step failed
}
  • The job is configured as a bunch of automation scripts running sequentially, e.g. ansible-playbook book.yml; python_script.py --switch; bash_script.sh; another_python.py --someotherswitch etc. If any one of these fails, the job goes red, obviously, so I had to pretty much work around every one of the issues that might occur, so none of the automation bits will do an exit(1). I do gather the results into a text file and cat it into the console, but what I'm really after is a way to have an indication of what failed right at the dashboard view and not in the console itself. – dyasny Dec 6 '16 at 1:33
  • Something like if some parts of the job completed OK, but some failed, to have the job marked as partial success, or in a yellow colour maybe, with a details line saying where exactly things failed. Doesn't have to be Jenkins, any tool that is capable will do, as long as it's opensource – dyasny Dec 6 '16 at 1:35

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.