3

I've read up on config management/provisioning tools like Ansible and SaltStack. These sound pretty good to me and I intend to use one of the two heavily (Not yet decided which altough I'm leaning towards Ansible). Ideally I want to use on of the two to control all aspects of configuration and command-execution in the system. I.e: from initial bootstrapping, ad-hoc commands, but also when a system-wide exception happens.

To this end it seems that I could use Nagios event handlers (when correctly setup) to in turn execute configured ansible playbooks (or the saltstack equivalant) to try to bring the system back in a correct state.

Is this setup often used? Any reasons this would not be a good idea?

I'm asking because it seems logical/convenient to me to have all config under 1 tool (ansible or saltstack), but information on using a combination of Nagios (or similar) and Ansible (or similar) as described seems to be really sparse/non-existent.

4
  • 1
    Why both Ansible and Salt? I'm under the impression that Ansible contains all of the functionality of Salt, but without having to install that pesky minion. Jul 15, 2013 at 9:59
  • Indeed, that wasn't clear from the question. Edited.
    – Gbrits
    Jul 15, 2013 at 10:02
  • 2
    I spent all weekend learning Ansible (it was too hot to go play in the sunshine). I'm very impressed generally. Jul 15, 2013 at 10:13
  • this is one of the big use cases for orchestration and runbook automation tools like Ansible, HP Operations Orchestration, BMC Atrium Orchestrator, etc. HP calls it "CLIP - Closed Loop Incident Process"
    – warren
    Jan 29, 2014 at 16:26

1 Answer 1

3

It's a reasonable idea, but you have to be VERY careful that your automated actions are precise and accurate.

You need to be absolutely sure that the failure state you're experiencing can be resolved with those automated actions to reset it (accurate).

You also need to make damn sure that your actions are totally idempotent, in case something goes wrong, and it triggers the wrong reconfiguration service. (precise).

It's not a bad idea, all together, but the problems surrounding state flap are the ones that'll catch you the quickest. From experience, they even catch me, where I've restarted something automatically, and not been aware that it's stuck (or running, depending on your point of view).

1
  • Agreed. Coming from a software-engineering background, I directly have to think of state-machines (or something more formal like petrinets) to describe all possible states and the exact ways in which one state can lead to another. This reduces complexity a lot when implemented correctly. I'm not sure though if something like this has entered the realm of sysadmin or if it's even feasible. If I'm going to setup a 'Nagios > ansible feedback loop' the first simple rule would be that only 1 state-changing command may be executed at a time. Otherwise state is bound to get undefined due to racing.
    – Gbrits
    Jul 15, 2013 at 10:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .