So first, I want to thank jscott, Fredi, user378016, and coderanger for all there answers.
To answer my own question
- For CF-Engine this isn't an issue as every node can be configured to run as a server and the runs will continue if the server isn't
This is all well documented on the CF-Engine website an example can be found here: https://cfengine.com/learn/how-cfengine-works/
- For Puppet you have a choice of Master/Masterless modes and their pro's and cons.
Puppet has a variety of modes and as Fredi has indicated the mode is one or the other. However after doing more digging Puppet is very flexible and has good features that can be supported for both master and masterless mode.
- For Chef the initial run requires the master server to fetch the policy but after that any run will continue with that current policy
if the master isn't available.
So this wasn't quite correct, when running is server mode (not using chef-solo) a run requires a connection to the master for each run. As has been mentioned there are ways to do fall back caching that has some interesting potential and maybe worth looking into some more.
- For Salt if the master server goes down then configuration is not enforced as all actions are done on the master
So thanks to user378016 for confirming, I think the answer provided does answer this quite nicely (permalink: https://serverfault.com/a/805791/225383)
- For Ansible (like salt) if the master server goes down then configurations are no longer enforced, as again all actions are done
by the master servers
So Ansible is a tricky one (again thanks to Fredi for his answer). It gives the strong benefit of only having to install the Ansible software on one server. The playbooks that stored on this "master" don't necessarily run on the master but can be applied on others via SSH. This of course requires all these boxes that you wish to configure being accessible via SSH and meet certain pre-conditions (as outlined in a playbook). In certain cases is not always that desirable.
Edit: I should add Ansible can run in a way that is similar to the puppet-masterless, chef-solo, by installing Ansible on the node to be managed and having it pull the configuration from somewhere like git and then apply the configuration locally.
For those that are interested in the direction I'm going in recommending CF-Engine, Chef or Puppet. While Ansible and Salt are both interesting, for my user case however it is not the optimal solution as I need to be able to ensure that policy's are enforced no matter what with good reporting metrics a reliance that SSH is always available is not really an option (and yes while we could install the server components on every box or a scheduler of some sort to force a configuration this seems counter intuitive to their original architecture to begin with).
All of these products are very good , and have different strengths and weaknesses, in this case I felt that Ansible and Salt are not suitable not only for the reason above but also for various other reasons as well.