Salt State Testing via Jenkins

Question

All,

We're trying to have automated tests in our Jenkins setup to run "smoke" and "lint" type of tests in our salt state files (.sls). All google-foo so far has yielded very little information. There is a way to test via test=True in the command line, but that doesn't work with a no-shell account (as Jenkin's accounts usually are).

I have not yet bumped into anyone doing this sort of automated testing of SaltStack states. So:

1) Is it possible

2) Anyone know of a good resource where I could look at

TIA.

Answer 1

Docker. Quick automated testing of server configuration is an undeniable real-world problem that docker nails. It can provide a clean computer already booted and listening on the network in a second. Start an image with /srv/salt bind-mounted and you can run salt-call --local state.highstate -l debug to test states without fussing with salt-key.

I know SaltStack, Inc used LXC much the same way. They likely still do.

As for the test - if you are clever and careful with your states files you could consider a clean second run to be an indication of success.

This is difficult to achieve as some states will always re-run. Salt Stack has been good at fixing these states as they are found. In the meantime you will have to surround these states with inline jinja conditionals that execute commands on the minion at runtime:

{% if salt['cmd.retcode']('your test here') %} 
some-identifier:
  some.module:
    - name: some anme
{% endif %}'

There exists a jenkins-docker plugin:

The aim of the docker plugin is to be able to use a docker host to dynamically provision a slave, run a single build, then tear-down that slave.

Alternatively, you can automate the whole thing via the new docker-ng salt module:

salt dockhost docker-ng.create states-qa rm=True binds="/srv/salt:/srv/salt"
salt dockhost docker-ng.retcode states-qa 'salt-call --local state.highstate' # run 1
salt dockhost docker-ng.retcode states-qa 'salt-call --local state.highstate' # run 2
salt dockhost docker-ng.stop states-qa

Answer 2

You might want to take a look at TestInfra

Answer 3

I've been searching a while for a good way to achieve this QA stuff on salt state, and my best answer so far is :

1. Using jenkins to launch jobs (through ssh), based on a dev git branch that:

   • Provision a lxc on our lab proxmox private cloud (in the exact same manner we do this in prod)

   • Using salt reactors the container get its config (as it would on prod)

   • Using testinfra to run unit test on the built and config'ed container

   • Finally if everything went OK destroy the container, if not keep it alive for a morning debugging session :)

2. We also run a linting jenkins jobs as :

   for state in $(sudo /usr/bin/salt-call cp.list_states | awk '{print $2}' | grep -v "^top$"); do sudo /usr/bin/salt-call --retcode-passthrough state.show_sls ${state} ; done
   

I still have some issue getting the correct return code for this last linting job (because of ssh and so on).

This process as a whole ensure :

1. Our provisioning process is OK
2. Our code base (state + pillar) is working as expected
3. We can merge dev to prod with a great confidence ratio

The real good point of testinfra is that it can use a salt connection backend which allow testinfra to connect to container without the need to deploy ssh key, or anything else (as we are using salt-cloud for the initial provisioning)

More on testinfra salt connection backend, testinfra salt module.

This not perfect but still it does quite a good job.